Note: This is test shard 3 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest
[ RUN ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 07:58:08.684950 16531 test_util.cc:268] Using random seed: -155224210
I20240502 07:58:08.692220 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:58:08.692297 16531 ts_itest-base.cc:113] --------------
I20240502 07:58:08.692315 16531 ts_itest-base.cc:114] 8 tablet servers
I20240502 07:58:08.692337 16531 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:58:08.692351 16531 ts_itest-base.cc:116] --------------
2024-05-02T07:58:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:58:08Z Disabled control of system clock
I20240502 07:58:08.702764 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:45191
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:45191
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 07:58:08.816202 16543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:08.816478 16543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:08.816560 16543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:08.819823 16543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:08.819916 16543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:58:08.819941 16543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:08.819958 16543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:58:08.819976 16543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:58:08.823993 16543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:45191
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.16543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:08.825057 16543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:08.826328 16543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:08.835425 16548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.836126 16551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:08.836843 16549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:08.837730 16543 server_base.cc:1034] running on GCE node
I20240502 07:58:08.838389 16543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:08.839473 16543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:08.840672 16543 hybrid_clock.cc:648] HybridClock initialized: now 1714636688840657 us; error 37 us; skew 500 ppm
I20240502 07:58:08.848780 16543 webserver.cc:460] Webserver started at http://127.16.36.254:35831/ using document root <none> and password file <none>
I20240502 07:58:08.849455 16543 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:08.849527 16543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:08.849732 16543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:08.851619 16543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "66a6984cb686449d8d32f27dbbe417ec"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-9sjq"
I20240502 07:58:08.852164 16543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "66a6984cb686449d8d32f27dbbe417ec"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-9sjq"
I20240502 07:58:08.856081 16543 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:08.858942 16557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:08.860226 16543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:08.860392 16543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "66a6984cb686449d8d32f27dbbe417ec"
format_stamp: "Formatted at 2024-05-02 07:58:08 on dist-test-slave-9sjq"
I20240502 07:58:08.860561 16543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:08.881800 16543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:08.882567 16543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:08.882763 16543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:08.890203 16543 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:45191
I20240502 07:58:08.890205 16609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:45191 every 8 connection(s)
I20240502 07:58:08.891356 16543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:58:08.894625 16610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:08.899688 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16543
I20240502 07:58:08.899837 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:58:08.900399 16610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec: Bootstrap starting.
I20240502 07:58:08.903194 16610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:08.904333 16610 log.cc:824] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:08.906266 16610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec: No bootstrap required, opened a new log
I20240502 07:58:08.909091 16610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a6984cb686449d8d32f27dbbe417ec" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45191 } }
I20240502 07:58:08.909345 16610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:08.909426 16610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a6984cb686449d8d32f27dbbe417ec, State: Initialized, Role: FOLLOWER
I20240502 07:58:08.909876 16610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a6984cb686449d8d32f27dbbe417ec" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45191 } }
I20240502 07:58:08.910009 16610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:58:08.910082 16610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:58:08.910195 16610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:08.911053 16610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a6984cb686449d8d32f27dbbe417ec" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45191 } }
I20240502 07:58:08.911419 16610 leader_election.cc:302] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a6984cb686449d8d32f27dbbe417ec; no voters:
I20240502 07:58:08.911866 16610 leader_election.cc:288] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:58:08.912428 16615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:08.912744 16615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [term 1 LEADER]: Becoming Leader. State: Replica: 66a6984cb686449d8d32f27dbbe417ec, State: Running, Role: LEADER
I20240502 07:58:08.913151 16615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a6984cb686449d8d32f27dbbe417ec" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45191 } }
I20240502 07:58:08.913573 16610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:58:08.914939 16616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a6984cb686449d8d32f27dbbe417ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a6984cb686449d8d32f27dbbe417ec" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45191 } } }
I20240502 07:58:08.915088 16616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:08.915290 16617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a6984cb686449d8d32f27dbbe417ec. Latest consensus state: current_term: 1 leader_uuid: "66a6984cb686449d8d32f27dbbe417ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a6984cb686449d8d32f27dbbe417ec" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45191 } } }
I20240502 07:58:08.915390 16617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec [sys.catalog]: This master's current role is: LEADER
I20240502 07:58:08.915843 16624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:58:08.919821 16624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:58:08.926604 16624 catalog_manager.cc:1347] Generated new cluster ID: 49fec2a57e5d429fa3a24a53d24183e1
I20240502 07:58:08.926712 16624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:58:08.939893 16624 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:58:08.941020 16624 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:58:08.948879 16624 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 66a6984cb686449d8d32f27dbbe417ec: Generated new TSK 0
I20240502 07:58:08.949930 16624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:58:08.964936 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 07:58:09.100225 16634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.100533 16634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.100701 16634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.104017 16634 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.104111 16634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.104198 16634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 07:58:09.108245 16634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.16634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.109360 16634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.110597 16634 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.120649 16642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.120995 16639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.121944 16640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.122754 16634 server_base.cc:1034] running on GCE node
I20240502 07:58:09.123235 16634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.123899 16634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.125072 16634 hybrid_clock.cc:648] HybridClock initialized: now 1714636689125054 us; error 34 us; skew 500 ppm
I20240502 07:58:09.127339 16634 webserver.cc:460] Webserver started at http://127.16.36.193:39393/ using document root <none> and password file <none>
I20240502 07:58:09.128002 16634 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.128067 16634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.128257 16634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.130064 16634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.130566 16634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.134474 16634 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20240502 07:58:09.137168 16648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.138424 16634 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:09.138612 16634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.138729 16634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.152390 16634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.153174 16634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.153362 16634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.153970 16634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.154958 16634 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.155010 16634 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.155076 16634 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.155107 16634 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.164655 16634 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:37289
I20240502 07:58:09.165951 16634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:58:09.164676 16761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:37289 every 8 connection(s)
I20240502 07:58:09.173017 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16634
I20240502 07:58:09.173161 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:58:09.176045 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:09.178783 16762 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:09.179096 16762 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.179944 16762 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:09.182149 16574 ts_manager.cc:194] Registered new tserver with Master: 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:09.184159 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:58599
W20240502 07:58:09.295469 16765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.295737 16765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.295785 16765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.299355 16765 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.299455 16765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.299533 16765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 07:58:09.303529 16765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.16765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.304580 16765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.305759 16765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.315356 16773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.315898 16770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.316830 16771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.317998 16765 server_base.cc:1034] running on GCE node
I20240502 07:58:09.318518 16765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.319228 16765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.320421 16765 hybrid_clock.cc:648] HybridClock initialized: now 1714636689320367 us; error 70 us; skew 500 ppm
I20240502 07:58:09.322698 16765 webserver.cc:460] Webserver started at http://127.16.36.194:45881/ using document root <none> and password file <none>
I20240502 07:58:09.323319 16765 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.323385 16765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.323576 16765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.325497 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.326021 16765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.329692 16765 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 07:58:09.332221 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.333463 16765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:58:09.333606 16765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.333714 16765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.347684 16765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.348542 16765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.348733 16765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.349368 16765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.350374 16765 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.350428 16765 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.350494 16765 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.350520 16765 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.360163 16765 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:37567
I20240502 07:58:09.360250 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:37567 every 8 connection(s)
I20240502 07:58:09.361317 16765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:58:09.363430 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16765
I20240502 07:58:09.363541 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:58:09.366437 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:09.371690 16893 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:09.372028 16893 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.372946 16893 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:09.374511 16574 ts_manager.cc:194] Registered new tserver with Master: ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:09.375219 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:37599
W20240502 07:58:09.488168 16896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.488432 16896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.488507 16896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.491791 16896 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.491881 16896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.491968 16896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 07:58:09.496017 16896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.16896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.497117 16896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.498325 16896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.507923 16901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.508394 16902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.509312 16904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.510128 16896 server_base.cc:1034] running on GCE node
I20240502 07:58:09.510596 16896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.511333 16896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.512553 16896 hybrid_clock.cc:648] HybridClock initialized: now 1714636689512506 us; error 67 us; skew 500 ppm
I20240502 07:58:09.514899 16896 webserver.cc:460] Webserver started at http://127.16.36.195:40957/ using document root <none> and password file <none>
I20240502 07:58:09.515520 16896 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.515586 16896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.515833 16896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.517714 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c24fcade15af45a79095717955b51756"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.518249 16896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c24fcade15af45a79095717955b51756"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.521891 16896 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:58:09.524546 16910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.525739 16896 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:09.525869 16896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c24fcade15af45a79095717955b51756"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.525969 16896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.539111 16896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.539873 16896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.540058 16896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.540694 16896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.541673 16896 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.541726 16896 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.541791 16896 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.541819 16896 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.551597 16896 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:38467
I20240502 07:58:09.551613 17023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:38467 every 8 connection(s)
I20240502 07:58:09.552701 16896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:58:09.562662 17024 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:09.562948 17024 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.562986 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16896
I20240502 07:58:09.563084 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:58:09.563614 17024 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:09.564970 16574 ts_manager.cc:194] Registered new tserver with Master: c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:09.565464 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:09.565739 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:59331
W20240502 07:58:09.682500 17027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.682768 17027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.682818 17027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.685930 17027 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.686008 17027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.686081 17027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 07:58:09.689867 17027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.17027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.690915 17027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.692065 17027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.701500 17032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.701956 17035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.702831 17033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.703609 17027 server_base.cc:1034] running on GCE node
I20240502 07:58:09.704109 17027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.704794 17027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.705968 17027 hybrid_clock.cc:648] HybridClock initialized: now 1714636689705940 us; error 47 us; skew 500 ppm
I20240502 07:58:09.708261 17027 webserver.cc:460] Webserver started at http://127.16.36.196:45369/ using document root <none> and password file <none>
I20240502 07:58:09.708921 17027 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.708985 17027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.709177 17027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.710978 17027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "914d87eb1b3345fbbb28738ee49e1455"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.711491 17027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "914d87eb1b3345fbbb28738ee49e1455"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.715378 17027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 07:58:09.718209 17041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.719413 17027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:09.719553 17027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "914d87eb1b3345fbbb28738ee49e1455"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.719671 17027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.731572 17027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.732265 17027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.732432 17027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.733039 17027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.733969 17027 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.734027 17027 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.734086 17027 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.734113 17027 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.743132 17027 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:46525
I20240502 07:58:09.743151 17154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:46525 every 8 connection(s)
I20240502 07:58:09.744199 17027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:58:09.752794 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17027
I20240502 07:58:09.752936 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:58:09.753929 17155 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:09.754235 17155 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.755002 17155 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:09.755347 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:0
--local_ip_for_outbound_sockets=127.16.36.197
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:09.756194 16574 ts_manager.cc:194] Registered new tserver with Master: 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:09.756834 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:45473
W20240502 07:58:09.868686 17158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:09.868954 17158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:09.869036 17158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:09.872277 17158 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:09.872375 17158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:09.872541 17158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:58:09.876448 17158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.17158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:09.877524 17158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:09.878710 17158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:09.887953 17163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.888671 17164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:09.889550 17166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:09.890317 17158 server_base.cc:1034] running on GCE node
I20240502 07:58:09.890794 17158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:09.891611 17158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:09.892781 17158 hybrid_clock.cc:648] HybridClock initialized: now 1714636689892759 us; error 39 us; skew 500 ppm
I20240502 07:58:09.895313 17158 webserver.cc:460] Webserver started at http://127.16.36.197:34619/ using document root <none> and password file <none>
I20240502 07:58:09.896042 17158 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:09.896126 17158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:09.896346 17158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:09.898604 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6c62852374eb46219eba5d56e8564238"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.899192 17158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6c62852374eb46219eba5d56e8564238"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.903765 17158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 07:58:09.906250 17172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.907454 17158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:09.907572 17158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6c62852374eb46219eba5d56e8564238"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:09.907692 17158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:09.926525 17158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:09.927256 17158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:09.927434 17158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:09.928047 17158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:09.929154 17158 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:09.929212 17158 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.929260 17158 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:09.929291 17158 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:09.938571 17158 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:39833
I20240502 07:58:09.938603 17285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:39833 every 8 connection(s)
I20240502 07:58:09.939617 17158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:09.941931 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17158
I20240502 07:58:09.942026 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:58:09.944439 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.198:0
--local_ip_for_outbound_sockets=127.16.36.198
--webserver_interface=127.16.36.198
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:09.949776 17286 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:09.950071 17286 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:09.950709 17286 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:09.951947 16574 ts_manager.cc:194] Registered new tserver with Master: 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833)
I20240502 07:58:09.952584 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:32935
W20240502 07:58:10.061492 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:10.061815 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:10.061900 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:10.065335 17289 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:10.065449 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:10.065558 17289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.198
I20240502 07:58:10.069573 17289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.198
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.17289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.198
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:10.070657 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:10.071875 17289 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:10.081208 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:10.081787 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:10.082403 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:10.083453 17289 server_base.cc:1034] running on GCE node
I20240502 07:58:10.083952 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:10.084621 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:10.085809 17289 hybrid_clock.cc:648] HybridClock initialized: now 1714636690085795 us; error 32 us; skew 500 ppm
I20240502 07:58:10.088019 17289 webserver.cc:460] Webserver started at http://127.16.36.198:46429/ using document root <none> and password file <none>
I20240502 07:58:10.088703 17289 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:10.088773 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:10.089000 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:10.091079 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "e9db5c47c0884fca96b3baabba7666cd"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.091588 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "e9db5c47c0884fca96b3baabba7666cd"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.095489 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 07:58:10.098019 17303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.099172 17289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:10.099300 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e9db5c47c0884fca96b3baabba7666cd"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.099408 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:10.122411 17289 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:10.123190 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:10.123368 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:10.123967 17289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:10.125121 17289 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:10.125192 17289 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.125263 17289 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:10.125294 17289 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.134783 17289 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.198:35011
I20240502 07:58:10.134814 17416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.198:35011 every 8 connection(s)
I20240502 07:58:10.135838 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:58:10.141505 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17289
I20240502 07:58:10.141624 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:58:10.144156 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.199:0
--local_ip_for_outbound_sockets=127.16.36.199
--webserver_interface=127.16.36.199
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:10.146131 17417 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:10.146395 17417 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:10.147022 17417 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:10.148321 16574 ts_manager.cc:194] Registered new tserver with Master: e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011)
I20240502 07:58:10.149236 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.198:45115
I20240502 07:58:10.187855 16762 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
W20240502 07:58:10.270588 17420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:10.270874 17420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:10.270954 17420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:10.274315 17420 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:10.274420 17420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:10.274528 17420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.199
I20240502 07:58:10.278558 17420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.199
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.17420
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.199
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:10.279582 17420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:10.280854 17420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:10.291962 17426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:10.292606 17428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:10.293430 17425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:10.294247 17420 server_base.cc:1034] running on GCE node
I20240502 07:58:10.294728 17420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:10.295344 17420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:10.296578 17420 hybrid_clock.cc:648] HybridClock initialized: now 1714636690296549 us; error 46 us; skew 500 ppm
I20240502 07:58:10.299098 17420 webserver.cc:460] Webserver started at http://127.16.36.199:35639/ using document root <none> and password file <none>
I20240502 07:58:10.299783 17420 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:10.299861 17420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:10.300055 17420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:10.302086 17420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "9ad83459fbb549c1a820893b7513cc2e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.302639 17420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "9ad83459fbb549c1a820893b7513cc2e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.306607 17420 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 07:58:10.309787 17434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.311626 17420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:10.311882 17420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9ad83459fbb549c1a820893b7513cc2e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.312007 17420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:10.331553 17420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:10.332276 17420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:10.332500 17420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:10.333151 17420 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:10.334112 17420 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:10.334173 17420 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.334245 17420 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:10.334282 17420 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.344125 17420 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.199:45283
I20240502 07:58:10.344133 17547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.199:45283 every 8 connection(s)
I20240502 07:58:10.345245 17420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:58:10.352025 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17420
I20240502 07:58:10.352154 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 07:58:10.354816 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.200:0
--local_ip_for_outbound_sockets=127.16.36.200
--webserver_interface=127.16.36.200
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45191
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:10.355569 17548 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:10.355818 17548 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:10.356549 17548 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:10.357972 16574 ts_manager.cc:194] Registered new tserver with Master: 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283)
I20240502 07:58:10.358723 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.199:51583
I20240502 07:58:10.378401 16893 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
W20240502 07:58:10.470578 17551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:10.470860 17551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:10.470945 17551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:10.474232 17551 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:10.474336 17551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:10.474455 17551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.200
I20240502 07:58:10.478365 17551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.200
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.17551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.200
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:10.479394 17551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:10.480630 17551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:10.490010 17559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:10.490602 17556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:10.491389 17557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:10.492267 17551 server_base.cc:1034] running on GCE node
I20240502 07:58:10.492787 17551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:10.493432 17551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:10.494626 17551 hybrid_clock.cc:648] HybridClock initialized: now 1714636690494609 us; error 34 us; skew 500 ppm
I20240502 07:58:10.497072 17551 webserver.cc:460] Webserver started at http://127.16.36.200:44357/ using document root <none> and password file <none>
I20240502 07:58:10.497684 17551 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:10.497749 17551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:10.497934 17551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:58:10.499732 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "25ced22802d74d21a8a3dd2ee690463e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.500247 17551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "25ced22802d74d21a8a3dd2ee690463e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.504283 17551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240502 07:58:10.507031 17565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.508538 17551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:10.508678 17551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "25ced22802d74d21a8a3dd2ee690463e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:10.508781 17551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:10.531050 17551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:10.531759 17551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:10.531927 17551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:10.532524 17551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:10.533638 17551 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:10.533695 17551 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.533762 17551 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:10.533794 17551 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:10.542629 17551 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.200:43119
I20240502 07:58:10.542658 17678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.200:43119 every 8 connection(s)
I20240502 07:58:10.543653 17551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:58:10.552197 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17551
I20240502 07:58:10.552330 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 07:58:10.553365 17679 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:10.553761 17679 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:10.554558 17679 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:10.555634 16574 ts_manager.cc:194] Registered new tserver with Master: 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119)
I20240502 07:58:10.556213 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.200:54209
I20240502 07:58:10.567617 16531 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 07:58:10.568596 17024 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
I20240502 07:58:10.582743 16531 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 07:58:10.582880 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17158
I20240502 07:58:10.588584 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17289
I20240502 07:58:10.596436 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17420
I20240502 07:58:10.602308 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17551
I20240502 07:58:10.759220 17155 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
I20240502 07:58:14.360564 16574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60014:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:14.382175 16957 tablet_service.cc:1467] Processing CreateTablet for tablet e46524c6180a487a80e3a740844cccdd (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.382596 16695 tablet_service.cc:1467] Processing CreateTablet for tablet 28c98a00679a446e995689f3cea9e91e (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.382596 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 28c98a00679a446e995689f3cea9e91e (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.382596 17089 tablet_service.cc:1467] Processing CreateTablet for tablet e46524c6180a487a80e3a740844cccdd (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.382756 16696 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.383292 16957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46524c6180a487a80e3a740844cccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.383389 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c98a00679a446e995689f3cea9e91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.383505 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c98a00679a446e995689f3cea9e91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.383786 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccdfb7bba3b42edae3a322bcd77ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.382176 16958 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.388825 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccdfb7bba3b42edae3a322bcd77ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.391314 17700 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:14.391378 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46524c6180a487a80e3a740844cccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.391410 17699 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:14.392014 17701 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:14.393805 17699 tablet_bootstrap.cc:654] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.393805 17700 tablet_bootstrap.cc:654] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.394560 17701 tablet_bootstrap.cc:654] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.394982 17700 log.cc:824] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:14.395448 17701 log.cc:824] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:14.396003 17699 log.cc:824] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:14.397182 17700 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:14.397388 17700 ts_tablet_manager.cc:1367] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:14.398046 17701 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:14.398258 17701 ts_tablet_manager.cc:1367] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:14.398691 17699 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:14.398342 16826 tablet_service.cc:1467] Processing CreateTablet for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.398864 17699 ts_tablet_manager.cc:1367] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 07:58:14.399343 16826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccdfb7bba3b42edae3a322bcd77ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.401989 17700 raft_consensus.cc:348] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.402034 17701 raft_consensus.cc:348] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.402254 17700 raft_consensus.cc:374] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.402256 17701 raft_consensus.cc:374] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.402319 17701 raft_consensus.cc:729] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.402318 17700 raft_consensus.cc:729] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.402638 17699 raft_consensus.cc:348] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.402820 17699 raft_consensus.cc:374] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.402861 17699 raft_consensus.cc:729] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.402788 17700 consensus_queue.cc:260] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.402802 17701 consensus_queue.cc:260] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.402886 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 28c98a00679a446e995689f3cea9e91e (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.403198 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28c98a00679a446e995689f3cea9e91e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.403245 17699 consensus_queue.cc:260] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.403736 17700 ts_tablet_manager.cc:1398] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240502 07:58:14.403738 17701 ts_tablet_manager.cc:1398] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:58:14.404140 17701 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:14.405202 17701 tablet_bootstrap.cc:654] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.403949 17699 ts_tablet_manager.cc:1398] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:58:14.406342 17699 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:14.407539 16827 tablet_service.cc:1467] Processing CreateTablet for tablet e46524c6180a487a80e3a740844cccdd (DEFAULT_TABLE table=rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.407620 17699 tablet_bootstrap.cc:654] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.407722 17700 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:14.407835 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46524c6180a487a80e3a740844cccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.408824 17700 tablet_bootstrap.cc:654] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.408938 17699 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:14.409042 17699 ts_tablet_manager.cc:1367] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:14.409660 17699 raft_consensus.cc:348] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.409801 17699 raft_consensus.cc:374] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.409842 17699 raft_consensus.cc:729] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.409973 17699 consensus_queue.cc:260] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.410292 17699 ts_tablet_manager.cc:1398] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:14.410483 17700 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:14.410580 17700 ts_tablet_manager.cc:1367] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:14.411178 17700 raft_consensus.cc:348] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.411314 17700 raft_consensus.cc:374] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.411347 17700 raft_consensus.cc:729] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.411476 17700 consensus_queue.cc:260] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.411749 17700 ts_tablet_manager.cc:1398] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:14.412525 17711 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:14.414976 17701 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:14.414973 17711 tablet_bootstrap.cc:654] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.415094 17701 ts_tablet_manager.cc:1367] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 07:58:14.415654 17701 raft_consensus.cc:348] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.415772 17701 raft_consensus.cc:374] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.415817 17701 raft_consensus.cc:729] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.415963 17701 consensus_queue.cc:260] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.416126 17711 log.cc:824] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:14.416261 17701 ts_tablet_manager.cc:1398] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:14.418170 17711 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:14.418386 17711 ts_tablet_manager.cc:1367] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:58:14.421021 17711 raft_consensus.cc:348] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.421245 17711 raft_consensus.cc:374] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.421324 17711 raft_consensus.cc:729] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.421741 17711 consensus_queue.cc:260] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.422607 17711 ts_tablet_manager.cc:1398] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 07:58:14.422909 17711 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:14.424028 17711 tablet_bootstrap.cc:654] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
W20240502 07:58:14.424407 16763 tablet.cc:2369] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:14.425390 17711 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:14.425463 17711 ts_tablet_manager.cc:1367] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:14.425976 17711 raft_consensus.cc:348] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.426098 17711 raft_consensus.cc:374] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.426127 17711 raft_consensus.cc:729] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.426270 17711 consensus_queue.cc:260] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.426523 17711 ts_tablet_manager.cc:1398] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:14.426688 17711 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:14.427657 17711 tablet_bootstrap.cc:654] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:14.428915 17711 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:14.428992 17711 ts_tablet_manager.cc:1367] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:14.429482 17711 raft_consensus.cc:348] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.429580 17711 raft_consensus.cc:374] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:14.429616 17711 raft_consensus.cc:729] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:14.429760 17711 consensus_queue.cc:260] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.430030 17711 ts_tablet_manager.cc:1398] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:14.451670 17706 raft_consensus.cc:482] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.451915 17706 raft_consensus.cc:504] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.452431 17713 raft_consensus.cc:482] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.452627 17713 raft_consensus.cc:504] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.453284 17706 leader_election.cc:288] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:14.453711 17713 leader_election.cc:288] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:14.458671 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:14.458673 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:14.458940 16847 raft_consensus.cc:2454] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 0.
I20240502 07:58:14.458966 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46524c6180a487a80e3a740844cccdd" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:14.459028 17109 raft_consensus.cc:2454] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 0.
I20240502 07:58:14.459208 16978 raft_consensus.cc:2454] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:14.459391 16650 leader_election.cc:302] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:14.459569 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46524c6180a487a80e3a740844cccdd" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:14.459600 16780 leader_election.cc:302] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:14.459676 17109 raft_consensus.cc:2454] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:14.459728 17706 raft_consensus.cc:2781] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.459802 17706 raft_consensus.cc:482] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.459851 17706 raft_consensus.cc:3037] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.459839 17713 raft_consensus.cc:2781] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.459898 17713 raft_consensus.cc:482] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.459937 17713 raft_consensus.cc:3037] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.460938 17713 raft_consensus.cc:504] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.460937 17706 raft_consensus.cc:504] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.461287 17713 leader_election.cc:288] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Requested vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:14.461308 17706 leader_election.cc:288] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:14.461611 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:14.461647 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46524c6180a487a80e3a740844cccdd" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:14.461728 16847 raft_consensus.cc:3037] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.461726 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:14.461753 16978 raft_consensus.cc:3037] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.461735 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e46524c6180a487a80e3a740844cccdd" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:14.461853 17109 raft_consensus.cc:3037] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.461864 17108 raft_consensus.cc:3037] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.462706 16847 raft_consensus.cc:2454] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 1.
I20240502 07:58:14.462842 16978 raft_consensus.cc:2454] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:14.462967 17109 raft_consensus.cc:2454] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 1.
I20240502 07:58:14.463042 17108 raft_consensus.cc:2454] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:14.463035 16650 leader_election.cc:302] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:14.463301 17706 raft_consensus.cc:2781] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.463490 16782 leader_election.cc:302] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:14.463670 17706 raft_consensus.cc:686] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 LEADER]: Becoming Leader. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Running, Role: LEADER
I20240502 07:58:14.463832 17713 raft_consensus.cc:2781] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.464023 17713 raft_consensus.cc:686] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Becoming Leader. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:14.464064 17706 consensus_queue.cc:237] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:14.464298 17713 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.467546 16574 catalog_manager.cc:5484] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:14.467546 16573 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:14.499339 17156 tablet.cc:2369] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:14.557814 17025 tablet.cc:2369] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:58:14.616568 16894 tablet.cc:2369] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:58:14.659545 17705 raft_consensus.cc:482] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:14.659757 17705 raft_consensus.cc:504] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.660971 17705 leader_election.cc:288] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:14.666239 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccdfb7bba3b42edae3a322bcd77ab86" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:14.666285 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccdfb7bba3b42edae3a322bcd77ab86" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:14.666445 16847 raft_consensus.cc:2454] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:14.666476 16716 raft_consensus.cc:2454] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:14.666802 16912 leader_election.cc:302] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:14.667052 17705 raft_consensus.cc:2781] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:14.667115 17705 raft_consensus.cc:482] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:14.667162 17705 raft_consensus.cc:3037] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.668022 17705 raft_consensus.cc:504] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.668367 17705 leader_election.cc:288] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Requested vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:14.668759 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccdfb7bba3b42edae3a322bcd77ab86" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:14.668869 16847 raft_consensus.cc:3037] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.669636 16847 raft_consensus.cc:2454] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:14.669903 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ccdfb7bba3b42edae3a322bcd77ab86" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:14.669963 16912 leader_election.cc:302] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:14.670047 16716 raft_consensus.cc:3037] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:14.670159 17705 raft_consensus.cc:2781] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:14.670514 17705 raft_consensus.cc:686] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Becoming Leader. State: Replica: c24fcade15af45a79095717955b51756, State: Running, Role: LEADER
I20240502 07:58:14.670913 16716 raft_consensus.cc:2454] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:14.670888 17705 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:14.674373 16573 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 reported cstate change: term changed from 0 to 1, leader changed from <none> to c24fcade15af45a79095717955b51756 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:14.793051 17730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:14.793329 17730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:14.796495 17730 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:58:14.909251 17108 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.910090 16978 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.911361 17722 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.911826 17713 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.914701 17109 raft_consensus.cc:1261] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer 9f88eac12f764ac7a1bb5a1bc9111d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.915230 17706 consensus_queue.cc:1035] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.917599 16847 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.918484 16716 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.919487 17705 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.919946 17705 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.920032 16847 raft_consensus.cc:1261] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer 9f88eac12f764ac7a1bb5a1bc9111d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:14.921211 17706 consensus_queue.cc:1035] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:14.926005 17763 mvcc.cc:205] Tried to move back new op lower bound from 7023151902338760704 to 7023151900528242688. Current Snapshot: MvccSnapshot[applied={T|T < 7023151902338760704}]
I20240502 07:58:14.930073 17761 mvcc.cc:205] Tried to move back new op lower bound from 7023151902364229632 to 7023151900528164864. Current Snapshot: MvccSnapshot[applied={T|T < 7023151902364229632}]
I20240502 07:58:14.932821 17759 mvcc.cc:205] Tried to move back new op lower bound from 7023151902356516864 to 7023151900528242688. Current Snapshot: MvccSnapshot[applied={T|T < 7023151902338760704}]
I20240502 07:58:14.938948 17768 mvcc.cc:205] Tried to move back new op lower bound from 7023151902364229632 to 7023151900528164864. Current Snapshot: MvccSnapshot[applied={T|T < 7023151902364229632}]
INSERT report
rows total: 2000
time total: 141.46 ms
time per row: 0.0707298 ms
I20240502 07:58:14.990931 16572 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60014:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:14.997277 16958 tablet_service.cc:1467] Processing CreateTablet for tablet 904dcca2b5be4bc5a225e201f647c0ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.997648 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904dcca2b5be4bc5a225e201f647c0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.997618 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab0feb808e741b0bd50990db02fab68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.997776 17089 tablet_service.cc:1467] Processing CreateTablet for tablet 904dcca2b5be4bc5a225e201f647c0ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.997849 16695 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab0feb808e741b0bd50990db02fab68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.997938 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab0feb808e741b0bd50990db02fab68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.998099 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab0feb808e741b0bd50990db02fab68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:14.998229 16696 tablet_service.cc:1467] Processing CreateTablet for tablet 904dcca2b5be4bc5a225e201f647c0ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:14.998456 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904dcca2b5be4bc5a225e201f647c0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.000702 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904dcca2b5be4bc5a225e201f647c0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.000892 17792 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:15.001240 16957 tablet_service.cc:1467] Processing CreateTablet for tablet c6e66836a49d42b48d1c9be0090565bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.001487 16957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e66836a49d42b48d1c9be0090565bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.001507 17790 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:15.001999 17792 tablet_bootstrap.cc:654] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.002393 17790 tablet_bootstrap.cc:654] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.003089 17791 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:15.003291 16827 tablet_service.cc:1467] Processing CreateTablet for tablet c6e66836a49d42b48d1c9be0090565bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.000881 16958 tablet_service.cc:1467] Processing CreateTablet for tablet 0ab0feb808e741b0bd50990db02fab68 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.003547 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e66836a49d42b48d1c9be0090565bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.003734 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab0feb808e741b0bd50990db02fab68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.004205 17791 tablet_bootstrap.cc:654] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.005162 16696 tablet_service.cc:1467] Processing CreateTablet for tablet c6e66836a49d42b48d1c9be0090565bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.005414 17791 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:15.005443 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e66836a49d42b48d1c9be0090565bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.005507 17791 ts_tablet_manager.cc:1367] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 07:58:15.005973 17796 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:15.005990 17791 raft_consensus.cc:348] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.006109 17791 raft_consensus.cc:374] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.006155 17791 raft_consensus.cc:729] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.006294 17791 consensus_queue.cc:260] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.006553 17791 ts_tablet_manager.cc:1398] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:15.006712 17791 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:15.007047 17796 tablet_bootstrap.cc:654] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.007568 17791 tablet_bootstrap.cc:654] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.008785 17791 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:15.008886 17791 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:15.009335 17796 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:15.009356 17791 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.009433 17796 ts_tablet_manager.cc:1367] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:15.009471 17791 raft_consensus.cc:374] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.009510 17791 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.009645 17791 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.009904 17791 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.010035 17796 raft_consensus.cc:348] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.010159 17796 raft_consensus.cc:374] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.010257 17796 raft_consensus.cc:729] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.010461 17796 consensus_queue.cc:260] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.010661 17790 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:15.010761 17790 ts_tablet_manager.cc:1367] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:58:15.010808 17796 ts_tablet_manager.cc:1398] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.010886 17792 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:15.010977 17792 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240502 07:58:15.011304 17790 raft_consensus.cc:348] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.011410 17790 raft_consensus.cc:374] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.011445 17790 raft_consensus.cc:729] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.011448 17792 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.011582 17792 raft_consensus.cc:374] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.011574 17790 consensus_queue.cc:260] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.011644 17792 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.011768 17792 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.011866 17790 ts_tablet_manager.cc:1398] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.012014 17792 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.012051 17790 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:15.012187 17792 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:15.013176 17790 tablet_bootstrap.cc:654] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.013645 17792 tablet_bootstrap.cc:654] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.014058 17767 raft_consensus.cc:482] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.014169 17767 raft_consensus.cc:504] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.015527 17767 leader_election.cc:288] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.016976 17790 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:15.017022 17792 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:15.017140 17792 ts_tablet_manager.cc:1367] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:15.017724 17792 raft_consensus.cc:348] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.017874 17792 raft_consensus.cc:374] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.017905 17792 raft_consensus.cc:729] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.018023 17792 consensus_queue.cc:260] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.018383 17792 ts_tablet_manager.cc:1398] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:15.018548 17792 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:15.018532 17790 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 07:58:15.019075 17790 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.019213 17790 raft_consensus.cc:374] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.019248 17790 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.019383 17790 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.019604 17792 tablet_bootstrap.cc:654] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.020089 17790 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:15.020311 17790 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:15.021245 17792 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:15.021350 17792 ts_tablet_manager.cc:1367] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:15.021505 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:15.021646 16716 raft_consensus.cc:2454] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 0.
I20240502 07:58:15.021883 17790 tablet_bootstrap.cc:654] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.021864 17792 raft_consensus.cc:348] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.021970 17792 raft_consensus.cc:374] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.022001 17792 raft_consensus.cc:729] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.022079 17044 leader_election.cc:302] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f; no voters:
I20240502 07:58:15.022135 17792 consensus_queue.cc:260] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.022403 17792 ts_tablet_manager.cc:1398] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:15.022617 17767 raft_consensus.cc:2781] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.022696 17767 raft_consensus.cc:482] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.022734 17767 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.023540 17790 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:15.023633 17790 ts_tablet_manager.cc:1367] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 07:58:15.024156 17767 raft_consensus.cc:504] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.024221 17790 raft_consensus.cc:348] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.024353 17790 raft_consensus.cc:374] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.024418 17790 raft_consensus.cc:729] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.024593 17767 leader_election.cc:288] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 election: Requested vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.024590 17790 consensus_queue.cc:260] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.025321 17790 ts_tablet_manager.cc:1398] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:15.025889 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:15.026005 16716 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.026847 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:15.026881 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:15.026973 16716 raft_consensus.cc:2454] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 1.
I20240502 07:58:15.026995 16978 raft_consensus.cc:2454] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 0.
I20240502 07:58:15.027338 17044 leader_election.cc:302] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f; no voters:
I20240502 07:58:15.027596 17767 raft_consensus.cc:2781] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.027792 17767 raft_consensus.cc:686] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Becoming Leader. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Running, Role: LEADER
I20240502 07:58:15.028118 17767 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.031148 16568 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:15.093791 17728 raft_consensus.cc:482] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.093947 17728 raft_consensus.cc:504] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.094372 17728 leader_election.cc:288] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.094898 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e66836a49d42b48d1c9be0090565bb" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:15.094945 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e66836a49d42b48d1c9be0090565bb" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:15.095079 16847 raft_consensus.cc:2454] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:15.095106 16716 raft_consensus.cc:2454] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:15.095472 16913 leader_election.cc:302] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:15.095737 17728 raft_consensus.cc:2781] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.095803 17728 raft_consensus.cc:482] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.095830 17728 raft_consensus.cc:3037] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.096757 17728 raft_consensus.cc:504] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.097141 17728 leader_election.cc:288] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.097589 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e66836a49d42b48d1c9be0090565bb" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:15.097592 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e66836a49d42b48d1c9be0090565bb" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:15.097736 16847 raft_consensus.cc:3037] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.097774 16716 raft_consensus.cc:3037] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.098759 16716 raft_consensus.cc:2454] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:15.098759 16847 raft_consensus.cc:2454] T c6e66836a49d42b48d1c9be0090565bb P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:15.099114 16913 leader_election.cc:302] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:15.099290 17728 raft_consensus.cc:2781] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.099380 17728 raft_consensus.cc:686] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Becoming Leader. State: Replica: c24fcade15af45a79095717955b51756, State: Running, Role: LEADER
I20240502 07:58:15.099534 17728 consensus_queue.cc:237] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.101471 16568 catalog_manager.cc:5484] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 reported cstate change: term changed from 0 to 1, leader changed from <none> to c24fcade15af45a79095717955b51756 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:15.444424 17783 raft_consensus.cc:482] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.444633 17783 raft_consensus.cc:504] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.445206 17783 leader_election.cc:288] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:15.445746 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:15.445932 17108 raft_consensus.cc:2454] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 0.
I20240502 07:58:15.446254 16651 leader_election.cc:302] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f; no voters:
I20240502 07:58:15.446466 17783 raft_consensus.cc:2781] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.446555 17783 raft_consensus.cc:482] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.446609 17783 raft_consensus.cc:3037] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.447567 17783 raft_consensus.cc:504] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.447991 17783 leader_election.cc:288] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 election: Requested vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:15.448422 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:15.448602 17108 raft_consensus.cc:3037] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.449213 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:15.449252 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:15.449363 16978 raft_consensus.cc:2454] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 0.
I20240502 07:58:15.449527 17108 raft_consensus.cc:2454] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 1.
I20240502 07:58:15.449796 16651 leader_election.cc:302] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f; no voters: c24fcade15af45a79095717955b51756
I20240502 07:58:15.449970 17783 raft_consensus.cc:2781] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.450163 17783 raft_consensus.cc:686] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 LEADER]: Becoming Leader. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Running, Role: LEADER
I20240502 07:58:15.450354 17783 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:15.452085 16568 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:15.524021 17728 consensus_queue.cc:1035] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.528057 17728 consensus_queue.cc:1035] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 07:58:15.581485 17810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:15.581779 17810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:15.584992 17810 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:15.586021 16978 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.587464 17819 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.591547 17819 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 07:58:15.698294 17108 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer 9f88eac12f764ac7a1bb5a1bc9111d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.698691 16978 raft_consensus.cc:3037] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.699782 16978 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 9f88eac12f764ac7a1bb5a1bc9111d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:15.700412 17783 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:15.702059 17783 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 135.332 ms
time per row: 0.0676658 ms
I20240502 07:58:15.768049 16569 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60014:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:15.774636 16827 tablet_service.cc:1467] Processing CreateTablet for tablet 83decbfaed2a4187bc40c51c35d61747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.774729 17089 tablet_service.cc:1467] Processing CreateTablet for tablet 83decbfaed2a4187bc40c51c35d61747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.775002 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83decbfaed2a4187bc40c51c35d61747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.775044 17089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83decbfaed2a4187bc40c51c35d61747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.775312 16958 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d8cb7f2784b14a64cba5fb33811ab (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.775592 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8cb7f2784b14a64cba5fb33811ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.775622 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d8cb7f2784b14a64cba5fb33811ab (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.775835 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8cb7f2784b14a64cba5fb33811ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.777057 16696 tablet_service.cc:1467] Processing CreateTablet for tablet 83decbfaed2a4187bc40c51c35d61747 (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.777334 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83decbfaed2a4187bc40c51c35d61747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.777999 17087 tablet_service.cc:1467] Processing CreateTablet for tablet 995d54008b154b088b0d8a49e71c93ca (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.778177 17873 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:15.778244 17087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995d54008b154b088b0d8a49e71c93ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.779172 17873 tablet_bootstrap.cc:654] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.779609 17874 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:15.779772 16695 tablet_service.cc:1467] Processing CreateTablet for tablet 995d54008b154b088b0d8a49e71c93ca (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.780014 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995d54008b154b088b0d8a49e71c93ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.780335 17874 tablet_bootstrap.cc:654] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.781415 17873 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:15.781541 17873 ts_tablet_manager.cc:1367] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.782088 17873 raft_consensus.cc:348] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:15.782431 16826 tablet_service.cc:1467] Processing CreateTablet for tablet 5a8d8cb7f2784b14a64cba5fb33811ab (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.782474 17873 raft_consensus.cc:374] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.782527 17873 raft_consensus.cc:729] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.782563 17872 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:15.782687 16826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8cb7f2784b14a64cba5fb33811ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.782687 17873 consensus_queue.cc:260] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:15.783083 17873 ts_tablet_manager.cc:1398] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.783440 17872 tablet_bootstrap.cc:654] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.784795 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 995d54008b154b088b0d8a49e71c93ca (DEFAULT_TABLE table=rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:15.784862 17872 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:15.784943 17872 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:15.785064 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 995d54008b154b088b0d8a49e71c93ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:15.785820 17872 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.785971 17872 raft_consensus.cc:374] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.786026 17872 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.786191 17872 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.786974 17872 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:15.787159 17872 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:15.787572 17877 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:15.788168 17872 tablet_bootstrap.cc:654] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.788561 17877 tablet_bootstrap.cc:654] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.789816 17874 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:15.789876 17877 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:15.789927 17874 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 07:58:15.789959 17877 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:15.790010 17872 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:15.790081 17872 ts_tablet_manager.cc:1367] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.790380 17874 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.790392 17877 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.790479 17874 raft_consensus.cc:374] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.790500 17874 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.790496 17877 raft_consensus.cc:374] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.790534 17877 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.790500 17872 raft_consensus.cc:348] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:15.790592 17872 raft_consensus.cc:374] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.790627 17872 raft_consensus.cc:729] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.790620 17874 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.790663 17877 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.790792 17872 consensus_queue.cc:260] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:15.790917 17874 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:15.790942 17877 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:15.791049 17872 ts_tablet_manager.cc:1398] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.791103 17874 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:15.791111 17877 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:15.791193 17872 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:15.792012 17877 tablet_bootstrap.cc:654] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.792058 17874 tablet_bootstrap.cc:654] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.792979 17872 tablet_bootstrap.cc:654] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.794299 17877 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:15.794454 17877 ts_tablet_manager.cc:1367] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:15.794843 17874 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:15.794950 17874 ts_tablet_manager.cc:1367] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:15.795159 17877 raft_consensus.cc:348] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:15.795347 17877 raft_consensus.cc:374] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.795437 17877 raft_consensus.cc:729] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.795435 17874 raft_consensus.cc:348] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.795531 17874 raft_consensus.cc:374] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.795571 17874 raft_consensus.cc:729] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.795626 17877 consensus_queue.cc:260] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:15.795698 17874 consensus_queue.cc:260] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.795820 17872 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:15.795917 17872 ts_tablet_manager.cc:1367] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:15.795987 17877 ts_tablet_manager.cc:1398] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:15.796015 17874 ts_tablet_manager.cc:1398] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.796331 17877 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:15.796665 17872 raft_consensus.cc:348] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.796787 17872 raft_consensus.cc:374] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.796829 17872 raft_consensus.cc:729] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.797111 17872 consensus_queue.cc:260] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.797430 17872 ts_tablet_manager.cc:1398] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.797646 17877 tablet_bootstrap.cc:654] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:15.798733 17877 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:15.798805 17877 ts_tablet_manager.cc:1367] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:15.799202 17877 raft_consensus.cc:348] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.799299 17877 raft_consensus.cc:374] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:15.799327 17877 raft_consensus.cc:729] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:15.799477 17877 consensus_queue.cc:260] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.799849 17877 ts_tablet_manager.cc:1398] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:15.864769 17849 raft_consensus.cc:482] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.864910 17849 raft_consensus.cc:504] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.865492 17849 leader_election.cc:288] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.866044 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995d54008b154b088b0d8a49e71c93ca" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:15.866232 16716 raft_consensus.cc:2454] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 0.
I20240502 07:58:15.866577 17044 leader_election.cc:302] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f; no voters:
I20240502 07:58:15.866806 17849 raft_consensus.cc:2781] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.866883 17849 raft_consensus.cc:482] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.866937 17849 raft_consensus.cc:3037] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.868139 17849 raft_consensus.cc:504] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.868561 17849 leader_election.cc:288] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.869163 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995d54008b154b088b0d8a49e71c93ca" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:15.869303 16716 raft_consensus.cc:3037] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.870258 16716 raft_consensus.cc:2454] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 1.
I20240502 07:58:15.870584 17044 leader_election.cc:302] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f; no voters:
I20240502 07:58:15.870770 17849 raft_consensus.cc:2781] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.870883 17849 raft_consensus.cc:686] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Becoming Leader. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Running, Role: LEADER
I20240502 07:58:15.871043 17849 consensus_queue.cc:237] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.871083 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995d54008b154b088b0d8a49e71c93ca" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:15.871178 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "995d54008b154b088b0d8a49e71c93ca" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:15.871227 16847 raft_consensus.cc:2454] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 0.
I20240502 07:58:15.873003 16571 catalog_manager.cc:5484] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:15.921895 17713 raft_consensus.cc:482] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:15.922071 17713 raft_consensus.cc:504] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.922645 17713 leader_election.cc:288] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.923151 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:15.923318 17108 raft_consensus.cc:2454] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:15.923669 16782 leader_election.cc:302] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:15.923880 17713 raft_consensus.cc:2781] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:15.923949 17713 raft_consensus.cc:482] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:15.924005 17713 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.925020 17713 raft_consensus.cc:504] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.925423 17713 leader_election.cc:288] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Requested vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:15.925856 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:15.925987 17108 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:15.926925 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:15.926950 16715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:15.927098 17108 raft_consensus.cc:2454] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:15.927145 16716 raft_consensus.cc:2454] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:15.927405 16782 leader_election.cc:302] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:15.927579 17713 raft_consensus.cc:2781] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:15.927878 17713 raft_consensus.cc:686] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Becoming Leader. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:15.928040 17713 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:15.929824 16571 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.033835 17849 raft_consensus.cc:482] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.033988 17849 raft_consensus.cc:504] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.034479 17849 leader_election.cc:288] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:16.034912 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8cb7f2784b14a64cba5fb33811ab" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:16.034958 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8cb7f2784b14a64cba5fb33811ab" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:16.035082 16978 raft_consensus.cc:2454] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 0.
I20240502 07:58:16.035082 16846 raft_consensus.cc:2454] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 0.
I20240502 07:58:16.035439 17042 leader_election.cc:302] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.035622 17849 raft_consensus.cc:2781] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.035693 17849 raft_consensus.cc:482] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.035722 17849 raft_consensus.cc:3037] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.036633 17849 raft_consensus.cc:504] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.037034 17849 leader_election.cc:288] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:16.037456 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8cb7f2784b14a64cba5fb33811ab" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:16.037506 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a8d8cb7f2784b14a64cba5fb33811ab" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:16.037577 16846 raft_consensus.cc:3037] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.037642 16978 raft_consensus.cc:3037] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.038498 16846 raft_consensus.cc:2454] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 1.
I20240502 07:58:16.038516 16978 raft_consensus.cc:2454] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 1.
I20240502 07:58:16.038908 17042 leader_election.cc:302] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.039161 17849 raft_consensus.cc:2781] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.039239 17849 raft_consensus.cc:686] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Becoming Leader. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Running, Role: LEADER
I20240502 07:58:16.039357 17849 consensus_queue.cc:237] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.041294 16571 catalog_manager.cc:5484] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:16.191788 17892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:16.192071 17892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:16.195483 17892 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:58:16.281970 16978 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.282050 16715 raft_consensus.cc:1261] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.282619 16716 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.282727 17819 consensus_queue.cc:1035] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.283097 16846 raft_consensus.cc:3037] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.283717 16716 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.284024 16846 raft_consensus.cc:1261] T 995d54008b154b088b0d8a49e71c93ca P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.284112 17713 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.284324 17108 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.284889 17713 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.285156 17849 consensus_queue.cc:1035] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.285534 17849 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.288755 16847 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.291222 17819 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 112.209 ms
time per row: 0.0561045 ms
I20240502 07:58:16.354343 16574 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60014:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:16.360823 16695 tablet_service.cc:1467] Processing CreateTablet for tablet d42e7b3acd724bd3b47d1dcb2e763f3d (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.360823 16958 tablet_service.cc:1467] Processing CreateTablet for tablet d42e7b3acd724bd3b47d1dcb2e763f3d (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.361006 16825 tablet_service.cc:1467] Processing CreateTablet for tablet d42e7b3acd724bd3b47d1dcb2e763f3d (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.361227 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42e7b3acd724bd3b47d1dcb2e763f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.361274 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42e7b3acd724bd3b47d1dcb2e763f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.361330 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d42e7b3acd724bd3b47d1dcb2e763f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.362432 16826 tablet_service.cc:1467] Processing CreateTablet for tablet 55aa016d8d254a9e83f94222c92dfff5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.362798 16957 tablet_service.cc:1467] Processing CreateTablet for tablet adb927f30a0e453591069d77d90d3077 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.362964 16956 tablet_service.cc:1467] Processing CreateTablet for tablet 55aa016d8d254a9e83f94222c92dfff5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.363054 16957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adb927f30a0e453591069d77d90d3077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.363231 16956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aa016d8d254a9e83f94222c92dfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.363734 17087 tablet_service.cc:1467] Processing CreateTablet for tablet adb927f30a0e453591069d77d90d3077 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.364010 17087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adb927f30a0e453591069d77d90d3077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.364356 16826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aa016d8d254a9e83f94222c92dfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.366475 17088 tablet_service.cc:1467] Processing CreateTablet for tablet 55aa016d8d254a9e83f94222c92dfff5 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.366672 17950 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:16.366766 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aa016d8d254a9e83f94222c92dfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.369055 16696 tablet_service.cc:1467] Processing CreateTablet for tablet adb927f30a0e453591069d77d90d3077 (DEFAULT_TABLE table=rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.369349 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adb927f30a0e453591069d77d90d3077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.369863 17949 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:16.370514 17951 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:16.371127 17949 tablet_bootstrap.cc:654] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.371632 17951 tablet_bootstrap.cc:654] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.372614 17949 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:16.372709 17949 ts_tablet_manager.cc:1367] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 07:58:16.372773 17951 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:16.372859 17951 ts_tablet_manager.cc:1367] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:16.373139 17949 raft_consensus.cc:348] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.373246 17949 raft_consensus.cc:374] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.373359 17949 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.373256 17951 raft_consensus.cc:348] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.373450 17951 raft_consensus.cc:374] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.373493 17951 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.373553 17949 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.373603 17951 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.373893 17951 ts_tablet_manager.cc:1398] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.373895 17949 ts_tablet_manager.cc:1398] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:16.374061 17951 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:16.374061 17949 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:16.374944 17951 tablet_bootstrap.cc:654] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.374944 17949 tablet_bootstrap.cc:654] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.375820 17950 tablet_bootstrap.cc:654] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.376544 17953 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:16.376900 17951 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:16.377310 17949 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:16.377415 17949 ts_tablet_manager.cc:1367] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:58:16.377719 17953 tablet_bootstrap.cc:654] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.377820 17949 raft_consensus.cc:348] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.377933 17949 raft_consensus.cc:374] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.377965 17949 raft_consensus.cc:729] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.378086 17949 consensus_queue.cc:260] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.378365 17949 ts_tablet_manager.cc:1398] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:58:16.378526 17949 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:16.379374 17949 tablet_bootstrap.cc:654] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.380617 17949 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:16.380734 17949 ts_tablet_manager.cc:1367] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:16.381187 17949 raft_consensus.cc:348] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.381296 17949 raft_consensus.cc:374] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.381328 17949 raft_consensus.cc:729] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.381610 17951 ts_tablet_manager.cc:1367] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240502 07:58:16.382169 17951 raft_consensus.cc:348] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.382266 17951 raft_consensus.cc:374] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.382299 17951 raft_consensus.cc:729] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.382409 17949 consensus_queue.cc:260] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.382781 17949 ts_tablet_manager.cc:1398] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:16.383122 17953 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:16.383081 17951 consensus_queue.cc:260] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.383760 17951 ts_tablet_manager.cc:1398] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:16.383433 17953 ts_tablet_manager.cc:1367] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:16.384807 17953 raft_consensus.cc:348] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.384985 17953 raft_consensus.cc:374] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.385054 17953 raft_consensus.cc:729] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.385383 17953 consensus_queue.cc:260] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.385651 17950 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:16.385768 17950 ts_tablet_manager.cc:1367] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20240502 07:58:16.386256 17950 raft_consensus.cc:348] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.386373 17950 raft_consensus.cc:374] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.386408 17950 raft_consensus.cc:729] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.386533 17950 consensus_queue.cc:260] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.386797 17950 ts_tablet_manager.cc:1398] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.387073 17950 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:16.387231 17953 ts_tablet_manager.cc:1398] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:58:16.387455 17953 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:16.388098 17950 tablet_bootstrap.cc:654] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.389779 17950 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:16.389873 17950 ts_tablet_manager.cc:1367] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:58:16.390369 17950 raft_consensus.cc:348] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.390488 17950 raft_consensus.cc:374] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.390525 17950 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.390645 17950 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.390949 17950 ts_tablet_manager.cc:1398] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.391567 17953 tablet_bootstrap.cc:654] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.392908 17953 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:16.393002 17953 ts_tablet_manager.cc:1367] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:16.393383 17953 raft_consensus.cc:348] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.393489 17953 raft_consensus.cc:374] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.393525 17953 raft_consensus.cc:729] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.393662 17953 consensus_queue.cc:260] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.393913 17953 ts_tablet_manager.cc:1398] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.396701 17859 raft_consensus.cc:482] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.396842 17859 raft_consensus.cc:504] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.397241 17859 leader_election.cc:288] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:16.397583 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e7b3acd724bd3b47d1dcb2e763f3d" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:16.397711 16716 raft_consensus.cc:2454] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:16.397642 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e7b3acd724bd3b47d1dcb2e763f3d" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:16.397821 16846 raft_consensus.cc:2454] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:16.398001 16913 leader_election.cc:302] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.398172 17859 raft_consensus.cc:2781] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.398233 17859 raft_consensus.cc:482] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.398294 17859 raft_consensus.cc:3037] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.399116 17859 raft_consensus.cc:504] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.399492 17859 leader_election.cc:288] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Requested vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:16.399969 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e7b3acd724bd3b47d1dcb2e763f3d" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:16.399947 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d42e7b3acd724bd3b47d1dcb2e763f3d" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:16.400082 16846 raft_consensus.cc:3037] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.400079 16716 raft_consensus.cc:3037] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.401108 16846 raft_consensus.cc:2454] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:16.401108 16716 raft_consensus.cc:2454] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:16.401435 16913 leader_election.cc:302] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.401612 17859 raft_consensus.cc:2781] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.401839 17859 raft_consensus.cc:686] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Becoming Leader. State: Replica: c24fcade15af45a79095717955b51756, State: Running, Role: LEADER
I20240502 07:58:16.402000 17859 consensus_queue.cc:237] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.403688 16574 catalog_manager.cc:5484] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 reported cstate change: term changed from 0 to 1, leader changed from <none> to c24fcade15af45a79095717955b51756 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.447108 17859 raft_consensus.cc:482] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.447260 17859 raft_consensus.cc:504] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.447795 17859 leader_election.cc:288] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:16.448733 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adb927f30a0e453591069d77d90d3077" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:16.448902 16716 raft_consensus.cc:2454] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:16.449208 16913 leader_election.cc:302] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.449402 17859 raft_consensus.cc:2781] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.449462 17859 raft_consensus.cc:482] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.449518 17859 raft_consensus.cc:3037] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.450469 17859 raft_consensus.cc:504] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.450898 17859 leader_election.cc:288] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Requested vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:16.451355 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adb927f30a0e453591069d77d90d3077" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:16.451506 16716 raft_consensus.cc:3037] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.451974 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adb927f30a0e453591069d77d90d3077" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:16.452041 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adb927f30a0e453591069d77d90d3077" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:16.452111 17109 raft_consensus.cc:2454] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:16.452522 16716 raft_consensus.cc:2454] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:16.452811 16913 leader_election.cc:302] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters: 914d87eb1b3345fbbb28738ee49e1455
I20240502 07:58:16.452987 17859 raft_consensus.cc:2781] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.453085 17859 raft_consensus.cc:686] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Becoming Leader. State: Replica: c24fcade15af45a79095717955b51756, State: Running, Role: LEADER
I20240502 07:58:16.453224 17859 consensus_queue.cc:237] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:58:16.455040 16574 catalog_manager.cc:5484] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 reported cstate change: term changed from 0 to 1, leader changed from <none> to c24fcade15af45a79095717955b51756 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.516219 17948 raft_consensus.cc:482] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.516359 17948 raft_consensus.cc:504] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.516911 17948 leader_election.cc:288] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:16.517390 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:16.517388 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:16.517550 17108 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:16.517550 16978 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:16.517944 16780 leader_election.cc:302] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:16.518138 17948 raft_consensus.cc:2781] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.518205 17948 raft_consensus.cc:482] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.518242 17948 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.519146 17948 raft_consensus.cc:504] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.519510 17948 leader_election.cc:288] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Requested vote from peers c24fcade15af45a79095717955b51756 (127.16.36.195:38467), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:16.520011 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:16.520015 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:16.520159 17108 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.520159 16978 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.521246 16978 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:16.521241 17108 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:16.521622 16782 leader_election.cc:302] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:16.521853 17948 raft_consensus.cc:2781] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.521960 17948 raft_consensus.cc:686] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Becoming Leader. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:16.522118 17948 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.524067 16568 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:16.648788 17967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:16.649075 17967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:16.652287 17967 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 07:58:16.752905 16846 raft_consensus.cc:1261] T d42e7b3acd724bd3b47d1dcb2e763f3d P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.753115 16716 raft_consensus.cc:1261] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.753480 16715 raft_consensus.cc:1261] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.753612 17996 consensus_queue.cc:1035] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.753847 17859 consensus_queue.cc:1035] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.754539 17859 consensus_queue.cc:1035] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.763190 17108 raft_consensus.cc:3037] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.764365 17108 raft_consensus.cc:1261] T adb927f30a0e453591069d77d90d3077 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.767642 17859 consensus_queue.cc:1035] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.768294 16978 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.773013 17948 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:16.773188 17108 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:16.778227 17948 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.725 ms
time per row: 0.0623626 ms
I20240502 07:58:16.827383 16572 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60014:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:58:16.833331 16825 tablet_service.cc:1467] Processing CreateTablet for tablet 54e55a09645e41e69a086b6e13dc8858 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.833730 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e55a09645e41e69a086b6e13dc8858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.833663 16695 tablet_service.cc:1467] Processing CreateTablet for tablet 54e55a09645e41e69a086b6e13dc8858 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.833938 16695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e55a09645e41e69a086b6e13dc8858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.834453 17088 tablet_service.cc:1467] Processing CreateTablet for tablet c7cdc29160a9482d8108af570a5dd404 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.834734 17088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cdc29160a9482d8108af570a5dd404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.834692 16696 tablet_service.cc:1467] Processing CreateTablet for tablet c7cdc29160a9482d8108af570a5dd404 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.834918 16696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cdc29160a9482d8108af570a5dd404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.836236 17953 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:16.836306 16957 tablet_service.cc:1467] Processing CreateTablet for tablet 54e55a09645e41e69a086b6e13dc8858 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.837065 16694 tablet_service.cc:1467] Processing CreateTablet for tablet 543e8d886e14407f9e2468d4802dd3b3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.837179 16957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e55a09645e41e69a086b6e13dc8858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.837225 17953 tablet_bootstrap.cc:654] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.837297 16694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543e8d886e14407f9e2468d4802dd3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.838459 17953 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:16.838542 17953 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:16.839059 17949 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:16.839027 17953 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.839129 17953 raft_consensus.cc:374] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.839172 17953 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.839300 17953 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.839587 17953 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.839725 17951 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:16.839771 17953 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:16.839918 17949 tablet_bootstrap.cc:654] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.840658 17953 tablet_bootstrap.cc:654] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.840725 17951 tablet_bootstrap.cc:654] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.840991 17949 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:16.841058 17949 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:16.841477 17949 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.841536 17950 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:16.841590 17949 raft_consensus.cc:374] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.841629 17949 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.841778 17949 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.842048 17953 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:16.842068 17949 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.842124 17953 ts_tablet_manager.cc:1367] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:16.842584 17950 tablet_bootstrap.cc:654] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.842564 17953 raft_consensus.cc:348] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.842665 17953 raft_consensus.cc:374] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.842696 17953 raft_consensus.cc:729] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.842842 17953 consensus_queue.cc:260] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.834462 17087 tablet_service.cc:1467] Processing CreateTablet for tablet 543e8d886e14407f9e2468d4802dd3b3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.843204 17953 ts_tablet_manager.cc:1398] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.843299 17087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543e8d886e14407f9e2468d4802dd3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.843356 17953 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:58:16.844221 17953 tablet_bootstrap.cc:654] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.846715 16957 tablet_service.cc:1467] Processing CreateTablet for tablet 543e8d886e14407f9e2468d4802dd3b3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.846843 17950 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:16.846905 17953 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: No bootstrap required, opened a new log
I20240502 07:58:16.846954 17950 ts_tablet_manager.cc:1367] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:58:16.846992 17953 ts_tablet_manager.cc:1367] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:58:16.847015 16957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 543e8d886e14407f9e2468d4802dd3b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.847177 16825 tablet_service.cc:1467] Processing CreateTablet for tablet c7cdc29160a9482d8108af570a5dd404 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:58:16.847440 16825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7cdc29160a9482d8108af570a5dd404. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:16.847380 17950 raft_consensus.cc:348] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.847424 17953 raft_consensus.cc:348] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.847498 17950 raft_consensus.cc:374] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.847522 17953 raft_consensus.cc:374] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.847528 17950 raft_consensus.cc:729] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.847550 17953 raft_consensus.cc:729] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.847647 17950 consensus_queue.cc:260] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.847693 17953 consensus_queue.cc:260] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.847949 17950 ts_tablet_manager.cc:1398] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.848093 17950 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455: Bootstrap starting.
I20240502 07:58:16.848817 17953 ts_tablet_manager.cc:1398] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:58:16.849202 17950 tablet_bootstrap.cc:654] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.850111 17951 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:16.850215 17951 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240502 07:58:16.850663 17951 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.850792 17951 raft_consensus.cc:374] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.850838 17951 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.850955 17951 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.851226 17951 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.851426 17951 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09: Bootstrap starting.
I20240502 07:58:16.851871 17949 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756: Bootstrap starting.
I20240502 07:58:16.852918 17949 tablet_bootstrap.cc:654] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.852926 17951 tablet_bootstrap.cc:654] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:58:16.854102 17783 raft_consensus.cc:482] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.854214 17783 raft_consensus.cc:504] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.854584 17950 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455: No bootstrap required, opened a new log
I20240502 07:58:16.854646 17783 leader_election.cc:288] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:16.854673 17950 ts_tablet_manager.cc:1367] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:58:16.855126 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:16.855149 17950 raft_consensus.cc:348] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.855253 17950 raft_consensus.cc:374] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.855266 16846 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 0.
I20240502 07:58:16.855288 17950 raft_consensus.cc:729] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.855402 17950 consensus_queue.cc:260] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.855615 16650 leader_election.cc:302] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:16.855651 17950 ts_tablet_manager.cc:1398] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.855989 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:16.856129 16978 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 0.
I20240502 07:58:16.856261 17783 raft_consensus.cc:2781] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.856328 17783 raft_consensus.cc:482] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.856369 17783 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.857547 17783 raft_consensus.cc:504] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.857661 17951 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09: No bootstrap required, opened a new log
I20240502 07:58:16.857748 17951 ts_tablet_manager.cc:1367] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:16.857892 17783 leader_election.cc:288] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:16.858018 17949 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756: No bootstrap required, opened a new log
I20240502 07:58:16.858098 17949 ts_tablet_manager.cc:1367] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:58:16.858225 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:16.858310 16846 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.858251 17951 raft_consensus.cc:348] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.858348 17951 raft_consensus.cc:374] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.858378 17951 raft_consensus.cc:729] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.858502 17951 consensus_queue.cc:260] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.858587 17949 raft_consensus.cc:348] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.858691 17949 raft_consensus.cc:374] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:58:16.858722 17949 raft_consensus.cc:729] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c24fcade15af45a79095717955b51756, State: Initialized, Role: FOLLOWER
I20240502 07:58:16.858812 17951 ts_tablet_manager.cc:1398] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:16.858843 17949 consensus_queue.cc:260] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.859131 17949 ts_tablet_manager.cc:1398] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:16.859144 16846 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 1.
I20240502 07:58:16.859460 16650 leader_election.cc:302] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:16.859637 17783 raft_consensus.cc:2781] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.859719 17783 raft_consensus.cc:686] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 LEADER]: Becoming Leader. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Running, Role: LEADER
I20240502 07:58:16.859863 17783 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:16.860252 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:16.860357 16978 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.861311 16978 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f88eac12f764ac7a1bb5a1bc9111d7f in term 1.
I20240502 07:58:16.862674 16572 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.912585 17996 raft_consensus.cc:482] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.912735 17996 raft_consensus.cc:504] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.913267 17996 leader_election.cc:288] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:16.913743 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543e8d886e14407f9e2468d4802dd3b3" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:16.913792 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543e8d886e14407f9e2468d4802dd3b3" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:16.913925 16716 raft_consensus.cc:2454] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:16.913934 17108 raft_consensus.cc:2454] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 0.
I20240502 07:58:16.914268 16913 leader_election.cc:302] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.914461 17996 raft_consensus.cc:2781] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.914525 17996 raft_consensus.cc:482] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.914577 17996 raft_consensus.cc:3037] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.915576 17996 raft_consensus.cc:504] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.915988 17996 leader_election.cc:288] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Requested vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:16.916380 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543e8d886e14407f9e2468d4802dd3b3" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:16.916433 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "543e8d886e14407f9e2468d4802dd3b3" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:16.916541 16716 raft_consensus.cc:3037] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.916577 17108 raft_consensus.cc:3037] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.917560 16716 raft_consensus.cc:2454] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:16.917559 17108 raft_consensus.cc:2454] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c24fcade15af45a79095717955b51756 in term 1.
I20240502 07:58:16.917894 16913 leader_election.cc:302] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:16.918077 17996 raft_consensus.cc:2781] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.918174 17996 raft_consensus.cc:686] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Becoming Leader. State: Replica: c24fcade15af45a79095717955b51756, State: Running, Role: LEADER
I20240502 07:58:16.918344 17996 consensus_queue.cc:237] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:16.920250 16572 catalog_manager.cc:5484] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 reported cstate change: term changed from 0 to 1, leader changed from <none> to c24fcade15af45a79095717955b51756 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:16.987342 17948 raft_consensus.cc:482] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:58:16.987496 17948 raft_consensus.cc:504] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.987974 17948 leader_election.cc:288] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:16.988524 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cdc29160a9482d8108af570a5dd404" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
I20240502 07:58:16.988641 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cdc29160a9482d8108af570a5dd404" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" is_pre_election: true
I20240502 07:58:16.988708 17108 raft_consensus.cc:2454] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:16.988806 16716 raft_consensus.cc:2454] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 0.
I20240502 07:58:16.989063 16782 leader_election.cc:302] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:16.989238 17948 raft_consensus.cc:2781] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:58:16.989302 17948 raft_consensus.cc:482] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:58:16.989349 17948 raft_consensus.cc:3037] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.990258 17948 raft_consensus.cc:504] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.990593 17948 leader_election.cc:288] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Requested vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:16.991073 16716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cdc29160a9482d8108af570a5dd404" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
I20240502 07:58:16.991194 16716 raft_consensus.cc:3037] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.991212 17108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7cdc29160a9482d8108af570a5dd404" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:16.991348 17108 raft_consensus.cc:3037] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:58:16.992015 16716 raft_consensus.cc:2454] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:16.992120 17108 raft_consensus.cc:2454] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:16.992316 16782 leader_election.cc:302] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:16.992499 17948 raft_consensus.cc:2781] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:58:16.992599 17948 raft_consensus.cc:686] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Becoming Leader. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:16.992750 17948 consensus_queue.cc:237] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:16.994604 16572 catalog_manager.cc:5484] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } }
W20240502 07:58:17.119029 18017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.119314 18017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.122632 18017 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 07:58:17.219031 16846 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer 9f88eac12f764ac7a1bb5a1bc9111d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.219521 17783 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.221140 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 9f88eac12f764ac7a1bb5a1bc9111d7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.221391 16716 raft_consensus.cc:1261] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.221810 17996 consensus_queue.cc:1035] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.224344 17108 raft_consensus.cc:1261] T 543e8d886e14407f9e2468d4802dd3b3 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.224673 18009 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.228281 17996 consensus_queue.cc:1035] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.229498 17109 raft_consensus.cc:1261] T c7cdc29160a9482d8108af570a5dd404 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.230068 17948 consensus_queue.cc:1035] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:58:17.230214 16715 raft_consensus.cc:1261] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:58:17.230666 17948 consensus_queue.cc:1035] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 129.11 ms
time per row: 0.0645551 ms
I20240502 07:58:17.299816 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:39833
--local_ip_for_outbound_sockets=127.16.36.197
--tserver_master_addrs=127.16.36.254:45191
--webserver_port=34619
--webserver_interface=127.16.36.197
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 07:58:17.418661 18051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.418922 18051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:17.418970 18051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.422117 18051 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:17.422205 18051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:17.422279 18051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:58:17.426309 18051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:39833
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=34619
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.18051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:17.427248 18051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:17.428369 18051 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:17.437425 18059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:17.438026 18057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:17.438814 18056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:17.439791 18051 server_base.cc:1034] running on GCE node
I20240502 07:58:17.440310 18051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:17.441479 18051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:17.442827 18051 hybrid_clock.cc:648] HybridClock initialized: now 1714636697442810 us; error 41 us; skew 500 ppm
I20240502 07:58:17.445119 18051 webserver.cc:460] Webserver started at http://127.16.36.197:34619/ using document root <none> and password file <none>
I20240502 07:58:17.445729 18051 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:17.445811 18051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:17.449609 18051 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240502 07:58:17.451721 18065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.452848 18051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:58:17.452961 18051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6c62852374eb46219eba5d56e8564238"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:58:17.453428 18051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:17.466174 18051 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:17.466918 18051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:17.467090 18051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:17.467703 18051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:17.468678 18051 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:17.468730 18051 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.468813 18051 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:17.468839 18051 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.477437 18051 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:39833
I20240502 07:58:17.477463 18178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:39833 every 8 connection(s)
I20240502 07:58:17.478533 18051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:58:17.486260 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 18051
I20240502 07:58:17.486732 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.198:35011
--local_ip_for_outbound_sockets=127.16.36.198
--tserver_master_addrs=127.16.36.254:45191
--webserver_port=46429
--webserver_interface=127.16.36.198
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:17.488865 18179 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:17.489156 18179 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:17.489821 18179 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:17.491326 16574 ts_manager.cc:194] Re-registered known tserver with Master: 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833)
I20240502 07:58:17.491940 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:47699
W20240502 07:58:17.604260 18182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.604562 18182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:17.604621 18182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.607862 18182 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:17.607940 18182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:17.608016 18182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.198
I20240502 07:58:17.611948 18182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.198:35011
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.198
--webserver_port=46429
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.18182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.198
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:17.612977 18182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:17.614228 18182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:17.623662 18188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:17.624182 18190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:17.624908 18187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:17.625703 18182 server_base.cc:1034] running on GCE node
I20240502 07:58:17.626181 18182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:17.626796 18182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:17.627961 18182 hybrid_clock.cc:648] HybridClock initialized: now 1714636697627938 us; error 41 us; skew 500 ppm
I20240502 07:58:17.630224 18182 webserver.cc:460] Webserver started at http://127.16.36.198:46429/ using document root <none> and password file <none>
I20240502 07:58:17.630841 18182 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:17.631047 18182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:17.634907 18182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 07:58:17.637107 18196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.638336 18182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:58:17.638475 18182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "e9db5c47c0884fca96b3baabba7666cd"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:17.638943 18182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:17.660092 18182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:17.660878 18182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:17.661046 18182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:17.661612 18182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:17.662551 18182 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:17.662609 18182 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.662669 18182 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:17.662694 18182 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.671860 18182 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.198:35011
I20240502 07:58:17.671892 18309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.198:35011 every 8 connection(s)
I20240502 07:58:17.673015 18182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:58:17.673377 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 18182
I20240502 07:58:17.673837 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.199:45283
--local_ip_for_outbound_sockets=127.16.36.199
--tserver_master_addrs=127.16.36.254:45191
--webserver_port=35639
--webserver_interface=127.16.36.199
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:17.684414 18310 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:17.684731 18310 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:17.685472 18310 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:17.686709 16574 ts_manager.cc:194] Re-registered known tserver with Master: e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011)
I20240502 07:58:17.687287 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.198:56535
W20240502 07:58:17.798983 18312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.799248 18312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:17.799319 18312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.802543 18312 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:17.802634 18312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:17.802731 18312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.199
I20240502 07:58:17.806557 18312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.199:45283
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.199
--webserver_port=35639
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.18312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.199
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:17.807631 18312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:17.808848 18312 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:17.817917 18319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:17.818385 18321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:17.819309 18318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:17.819967 18312 server_base.cc:1034] running on GCE node
I20240502 07:58:17.820495 18312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:17.821144 18312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:17.822338 18312 hybrid_clock.cc:648] HybridClock initialized: now 1714636697822316 us; error 38 us; skew 500 ppm
I20240502 07:58:17.824651 18312 webserver.cc:460] Webserver started at http://127.16.36.199:35639/ using document root <none> and password file <none>
I20240502 07:58:17.825259 18312 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:17.825331 18312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:17.829104 18312 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20240502 07:58:17.831467 18327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.832746 18312 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:17.832861 18312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "9ad83459fbb549c1a820893b7513cc2e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:17.833284 18312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:17.846786 18312 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:17.847453 18312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:17.847604 18312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:17.848129 18312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:17.849098 18312 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:17.849150 18312 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.849192 18312 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:17.849206 18312 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:17.858325 18312 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.199:45283
I20240502 07:58:17.858386 18440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.199:45283 every 8 connection(s)
I20240502 07:58:17.859447 18312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:58:17.860499 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 18312
I20240502 07:58:17.860965 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.200:43119
--local_ip_for_outbound_sockets=127.16.36.200
--tserver_master_addrs=127.16.36.254:45191
--webserver_port=44357
--webserver_interface=127.16.36.200
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 07:58:17.870158 18441 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:17.870416 18441 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:17.871109 18441 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:17.872383 16574 ts_manager.cc:194] Re-registered known tserver with Master: 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283)
I20240502 07:58:17.873240 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.199:34135
W20240502 07:58:17.980363 18443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:17.980643 18443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:58:17.980701 18443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:17.984107 18443 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:58:17.984194 18443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:58:17.984269 18443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.200
I20240502 07:58:17.988513 18443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.200:43119
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.200
--webserver_port=44357
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.18443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.200
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:58:17.989491 18443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:58:17.990681 18443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:58:18.000098 18450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:18.000641 18449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:58:18.001447 18452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:58:18.002351 18443 server_base.cc:1034] running on GCE node
I20240502 07:58:18.002827 18443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:58:18.003479 18443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:58:18.004684 18443 hybrid_clock.cc:648] HybridClock initialized: now 1714636698004658 us; error 43 us; skew 500 ppm
I20240502 07:58:18.007292 18443 webserver.cc:460] Webserver started at http://127.16.36.200:44357/ using document root <none> and password file <none>
I20240502 07:58:18.008074 18443 fs_manager.cc:359] Metadata directory not provided
I20240502 07:58:18.008168 18443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:58:18.011783 18443 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:58:18.014034 18459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:18.015189 18443 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:58:18.015300 18443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "25ced22802d74d21a8a3dd2ee690463e"
format_stamp: "Formatted at 2024-05-02 07:58:10 on dist-test-slave-9sjq"
I20240502 07:58:18.015763 18443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:58:18.030084 18443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:58:18.030766 18443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:58:18.030916 18443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:58:18.031456 18443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:58:18.032372 18443 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:58:18.032419 18443 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:18.032490 18443 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:58:18.032524 18443 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:58:18.041730 18443 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.200:43119
I20240502 07:58:18.041755 18572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.200:43119 every 8 connection(s)
I20240502 07:58:18.042876 18443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:58:18.048573 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 18443
I20240502 07:58:18.052814 18573 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:58:18.053057 18573 heartbeater.cc:464] Registering TS with master...
I20240502 07:58:18.053735 18573 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:58:18.054894 16574 ts_manager.cc:194] Re-registered known tserver with Master: 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119)
I20240502 07:58:18.055465 16574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.200:38697
I20240502 07:58:18.259881 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.260370 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.268129 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.268936 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:18.280027 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.284265 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.288272 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:18.291481 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.484841 16574 ts_manager.cc:292] Set tserver state for 9f88eac12f764ac7a1bb5a1bc9111d7f to MAINTENANCE_MODE
I20240502 07:58:18.494750 18179 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
W20240502 07:58:18.617873 18637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:18.618142 18637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:18.621608 18637 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:18.689808 18310 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
I20240502 07:58:18.723475 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.725106 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.734040 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.734503 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.743160 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.750882 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.753899 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.758103 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.758869 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.760979 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.764088 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:18.770509 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.773459 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.794937 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:18.811906 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:18.825639 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:18.875538 18441 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
I20240502 07:58:18.952375 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.952543 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.955209 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:18.956811 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.961807 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:18.963486 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:18.971848 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.005574 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 07:58:19.043987 18733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:19.044368 18733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:19.048867 18733 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:19.057794 18573 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
I20240502 07:58:19.156018 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.162004 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.163319 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.170372 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.192744 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.197592 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.203397 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:19.209328 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.211256 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.216696 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.225118 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.228176 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:19.229615 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:19.243564 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.265190 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:19.282967 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.320313 16847 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } }
I20240502 07:58:19.322180 16716 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:19.322391 17109 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:19.322819 18844 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:19.323041 17889 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:19.325359 18844 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } }
I20240502 07:58:19.325739 17109 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } }
I20240502 07:58:19.328637 16574 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } }
I20240502 07:58:19.328228 16716 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } }
I20240502 07:58:19.334884 16846 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:19.336622 17109 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:19.336771 16716 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:19.337568 18844 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:19.337956 18844 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:19.339970 17889 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:19.340503 16716 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
W20240502 07:58:19.342666 16780 consensus_peers.cc:485] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833): Couldn't send request to peer 6c62852374eb46219eba5d56e8564238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83decbfaed2a4187bc40c51c35d61747. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:19.342783 16559 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 83decbfaed2a4187bc40c51c35d61747 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:58:19.343892 16569 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f reported cstate change: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:19.346376 17109 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
W20240502 07:58:19.434587 18836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:19.434895 18836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:58:19.438488 18836 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:19.529485 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.530601 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.533432 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.533869 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.536372 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:19.543264 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.548540 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.557080 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:19.724175 18899 ts_tablet_manager.cc:902] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:19.724987 18899 tablet_copy_client.cc:308] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:19.729283 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 83decbfaed2a4187bc40c51c35d61747 from peer 6c62852374eb46219eba5d56e8564238 ({username='slave'} at 127.16.36.197:35345)
I20240502 07:58:19.729481 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 83decbfaed2a4187bc40c51c35d61747 from peer 6c62852374eb46219eba5d56e8564238 at {username='slave'} at 127.16.36.197:35345: session id = 6c62852374eb46219eba5d56e8564238-83decbfaed2a4187bc40c51c35d61747
I20240502 07:58:19.731154 16867 tablet_copy_source_session.cc:205] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:19.734696 18899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83decbfaed2a4187bc40c51c35d61747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:19.741197 18899 tablet_copy_client.cc:783] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:19.741411 18899 tablet_copy_client.cc:647] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 1 WAL segments...
W20240502 07:58:19.742025 18898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:58:19.742290 18898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 07:58:19.743162 18903 tablet_copy_client.cc:1002] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:19.745826 18898 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:58:19.844297 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.844338 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.844758 17087 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.849623 16694 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:19.850315 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.851866 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:19.852560 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:19.853842 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Runtime error: ksck discovered errors
I20240502 07:58:19.913767 18635 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16634
W20240502 07:58:19.955170 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:19.956020 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:19.992527 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.089529 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.089673 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:20.090494 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.105605 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.284061 16782 consensus_peers.cc:485] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.316154 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:20.317232 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.422835 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:20.449960 17044 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:21.023828 18970 raft_consensus.cc:482] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.023970 18970 raft_consensus.cc:504] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:21.024447 18970 leader_election.cc:288] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:21.024863 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
W20240502 07:58:21.026060 16913 leader_election.cc:334] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.026129 16913 leader_election.cc:302] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c24fcade15af45a79095717955b51756; no voters: 9f88eac12f764ac7a1bb5a1bc9111d7f, ec22d92eae69437a92d6dc5a1ca43b09
I20240502 07:58:21.026278 18970 raft_consensus.cc:2726] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:58:21.057991 18855 raft_consensus.cc:482] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.058138 18855 raft_consensus.cc:504] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:21.058614 18855 leader_election.cc:288] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:21.059144 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:21.059332 16977 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 1.
I20240502 07:58:21.059651 16780 leader_election.cc:302] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
W20240502 07:58:21.059751 16782 leader_election.cc:334] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.059824 18855 raft_consensus.cc:2781] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:58:21.059878 18855 raft_consensus.cc:482] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.059926 18855 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:21.060853 18855 raft_consensus.cc:504] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:21.061223 18855 leader_election.cc:288] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 election: Requested vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:21.061609 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:21.061743 16977 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:21.062196 16782 leader_election.cc:334] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.062743 16977 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec22d92eae69437a92d6dc5a1ca43b09 in term 2.
I20240502 07:58:21.063033 16780 leader_election.cc:302] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters: 9f88eac12f764ac7a1bb5a1bc9111d7f
I20240502 07:58:21.063216 18855 raft_consensus.cc:2781] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:21.063314 18855 raft_consensus.cc:686] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Becoming Leader. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:21.063459 18855 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:21.065186 16571 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: term changed from 1 to 2, leader changed from 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) to ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194). New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:21.074388 18855 raft_consensus.cc:482] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.074512 18855 raft_consensus.cc:504] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.074865 18855 leader_election.cc:288] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:21.075523 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
W20240502 07:58:21.075839 16782 leader_election.cc:334] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.076021 16782 leader_election.cc:302] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec22d92eae69437a92d6dc5a1ca43b09; no voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f
I20240502 07:58:21.076170 18855 raft_consensus.cc:2726] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:58:21.146584 18970 raft_consensus.cc:482] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.146723 18970 raft_consensus.cc:504] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.147135 18970 leader_election.cc:288] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:21.147724 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "c24fcade15af45a79095717955b51756" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "914d87eb1b3345fbbb28738ee49e1455" is_pre_election: true
W20240502 07:58:21.148123 16913 leader_election.cc:334] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.148191 16913 leader_election.cc:302] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c24fcade15af45a79095717955b51756; no voters: 914d87eb1b3345fbbb28738ee49e1455, 9f88eac12f764ac7a1bb5a1bc9111d7f
I20240502 07:58:21.148303 18970 raft_consensus.cc:2726] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:58:21.213074 18971 raft_consensus.cc:482] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.213238 18971 raft_consensus.cc:504] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.213708 18971 leader_election.cc:288] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:21.214215 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" is_pre_election: true
I20240502 07:58:21.214404 16846 raft_consensus.cc:2454] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 1.
I20240502 07:58:21.214702 17043 leader_election.cc:302] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
W20240502 07:58:21.214926 17044 leader_election.cc:334] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.215006 18971 raft_consensus.cc:2781] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:58:21.215095 18971 raft_consensus.cc:482] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.215152 18971 raft_consensus.cc:3037] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:21.216096 18971 raft_consensus.cc:504] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.216421 18971 leader_election.cc:288] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:58:21.216853 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28c98a00679a446e995689f3cea9e91e" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:21.217001 16846 raft_consensus.cc:3037] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:21.217535 17044 leader_election.cc:334] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.218042 16846 raft_consensus.cc:2454] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 2.
I20240502 07:58:21.218338 17043 leader_election.cc:302] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, ec22d92eae69437a92d6dc5a1ca43b09; no voters: 9f88eac12f764ac7a1bb5a1bc9111d7f
I20240502 07:58:21.218493 18971 raft_consensus.cc:2781] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:21.218578 18971 raft_consensus.cc:686] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Becoming Leader. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Running, Role: LEADER
I20240502 07:58:21.218739 18971 consensus_queue.cc:237] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.220646 16571 catalog_manager.cc:5484] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: term changed from 1 to 2, leader changed from 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) to 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196). New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:21.243645 18899 tablet_copy_client.cc:523] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:21.245642 18899 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Bootstrap starting.
I20240502 07:58:21.251259 18899 log.cc:824] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:21.268031 18971 raft_consensus.cc:482] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.268190 18971 raft_consensus.cc:504] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.268680 18971 leader_election.cc:288] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:21.269142 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756" is_pre_election: true
I20240502 07:58:21.269326 16977 raft_consensus.cc:2454] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 1.
I20240502 07:58:21.269642 17042 leader_election.cc:302] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, c24fcade15af45a79095717955b51756; no voters:
I20240502 07:58:21.269834 18971 raft_consensus.cc:2781] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20240502 07:58:21.269860 17044 leader_election.cc:334] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.269896 18971 raft_consensus.cc:482] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9f88eac12f764ac7a1bb5a1bc9111d7f)
I20240502 07:58:21.269944 18971 raft_consensus.cc:3037] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:21.271025 18971 raft_consensus.cc:504] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.271449 18971 leader_election.cc:288] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 election: Requested vote from peers 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289), c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:21.271919 16977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ab0feb808e741b0bd50990db02fab68" candidate_uuid: "914d87eb1b3345fbbb28738ee49e1455" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:21.272073 16977 raft_consensus.cc:3037] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Advancing to term 2
W20240502 07:58:21.272516 17044 leader_election.cc:334] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:21.273042 16977 raft_consensus.cc:2454] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 914d87eb1b3345fbbb28738ee49e1455 in term 2.
I20240502 07:58:21.273346 17042 leader_election.cc:302] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 914d87eb1b3345fbbb28738ee49e1455, c24fcade15af45a79095717955b51756; no voters: 9f88eac12f764ac7a1bb5a1bc9111d7f
I20240502 07:58:21.273510 18971 raft_consensus.cc:2781] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:21.273703 18971 raft_consensus.cc:686] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Becoming Leader. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Running, Role: LEADER
I20240502 07:58:21.273883 18971 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:21.275234 18899 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:21.275593 18899 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Bootstrap complete.
I20240502 07:58:21.275915 18899 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20240502 07:58:21.275884 16571 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: term changed from 1 to 2, leader changed from 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) to 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196). New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:21.278508 18899 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:21.278774 18899 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Initialized, Role: LEARNER
I20240502 07:58:21.279134 18899 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:21.279938 18899 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
I20240502 07:58:21.280845 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session 6c62852374eb46219eba5d56e8564238-83decbfaed2a4187bc40c51c35d61747 received from {username='slave'} at 127.16.36.197:35345
I20240502 07:58:21.280934 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session 6c62852374eb46219eba5d56e8564238-83decbfaed2a4187bc40c51c35d61747 on tablet 83decbfaed2a4187bc40c51c35d61747 with peer 6c62852374eb46219eba5d56e8564238
W20240502 07:58:21.448185 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:21.514534 18133 raft_consensus.cc:1203] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 07:58:21.610548 16977 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:58:21.611090 18855 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:58:21.701931 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:21.746960 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:21.771415 16977 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:58:21.771934 18978 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:21.830916 16846 raft_consensus.cc:1261] T 28c98a00679a446e995689f3cea9e91e P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 07:58:21.831455 18971 consensus_queue.cc:1035] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:22.007926 18855 raft_consensus.cc:1050] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER 6c62852374eb46219eba5d56e8564238 to VOTER
I20240502 07:58:22.008370 18855 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:22.010007 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:22.010077 17109 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:22.010540 19025 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 07:58:22.010674 16782 consensus_peers.cc:485] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:22.010708 18855 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:22.014075 18855 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:22.014649 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:22.016049 17109 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:22.016844 16574 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:22.021029 16846 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:22.022378 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:22.022516 17109 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:22.022765 19009 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:22.022897 19025 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:22.024513 19025 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:22.025364 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:22.026047 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83decbfaed2a4187bc40c51c35d61747 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:22.026490 17109 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:22.027001 16571 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 07:58:22.030680 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:22.031677 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83decbfaed2a4187bc40c51c35d61747: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:22.267539 17109 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:22.269061 16977 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:22.269498 18978 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:58:22.269776 17044 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:22.271509 18971 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } }
I20240502 07:58:22.273326 16977 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } }
I20240502 07:58:22.273984 16565 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:22.278146 17109 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:22.280120 16977 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:58:22.280521 17044 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:22.280781 19054 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:22.282763 18978 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:22.283197 16977 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:22.284351 16560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 904dcca2b5be4bc5a225e201f647c0ea with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:58:22.285753 16565 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 4 to 5, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:22.286489 17042 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283): Couldn't send request to peer 9ad83459fbb549c1a820893b7513cc2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 904dcca2b5be4bc5a225e201f647c0ea. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:22.472877 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:22.554630 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:22.586789 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:22.632167 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:22.655105 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 07:58:22.731873 19067 ts_tablet_manager.cc:902] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Initiating tablet copy from peer 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:22.732702 19067 tablet_copy_client.cc:308] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:46525
I20240502 07:58:22.736582 17129 tablet_copy_service.cc:140] P 914d87eb1b3345fbbb28738ee49e1455: Received BeginTabletCopySession request for tablet 904dcca2b5be4bc5a225e201f647c0ea from peer 9ad83459fbb549c1a820893b7513cc2e ({username='slave'} at 127.16.36.199:52645)
I20240502 07:58:22.736867 17129 tablet_copy_service.cc:161] P 914d87eb1b3345fbbb28738ee49e1455: Beginning new tablet copy session on tablet 904dcca2b5be4bc5a225e201f647c0ea from peer 9ad83459fbb549c1a820893b7513cc2e at {username='slave'} at 127.16.36.199:52645: session id = 9ad83459fbb549c1a820893b7513cc2e-904dcca2b5be4bc5a225e201f647c0ea
I20240502 07:58:22.738255 17129 tablet_copy_source_session.cc:205] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:22.741627 19067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904dcca2b5be4bc5a225e201f647c0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:22.748373 19067 tablet_copy_client.cc:783] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:22.748761 19067 tablet_copy_client.cc:647] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:22.750555 19071 tablet_copy_client.cc:1002] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:22.811316 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:22.841881 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:23.243301 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83decbfaed2a4187bc40c51c35d61747: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:24.014034 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 07:58:24.251147 19067 tablet_copy_client.cc:523] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:24.253394 19067 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap starting.
I20240502 07:58:24.259239 19067 log.cc:824] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:24.279951 19067 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:24.280323 19067 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap complete.
I20240502 07:58:24.280658 19067 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Time spent bootstrapping tablet: real 0.027s user 0.027s sys 0.001s
I20240502 07:58:24.282753 19067 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:24.283025 19067 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ad83459fbb549c1a820893b7513cc2e, State: Initialized, Role: LEARNER
I20240502 07:58:24.283326 19067 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:24.284194 19067 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 07:58:24.285125 17129 tablet_copy_service.cc:342] P 914d87eb1b3345fbbb28738ee49e1455: Request end of tablet copy session 9ad83459fbb549c1a820893b7513cc2e-904dcca2b5be4bc5a225e201f647c0ea received from {username='slave'} at 127.16.36.199:52645
I20240502 07:58:24.285220 17129 tablet_copy_service.cc:433] P 914d87eb1b3345fbbb28738ee49e1455: ending tablet copy session 9ad83459fbb549c1a820893b7513cc2e-904dcca2b5be4bc5a225e201f647c0ea on tablet 904dcca2b5be4bc5a225e201f647c0ea with peer 9ad83459fbb549c1a820893b7513cc2e
W20240502 07:58:24.297335 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:24.312069 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83decbfaed2a4187bc40c51c35d61747: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:24.343987 18395 raft_consensus.cc:1203] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 07:58:24.469825 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:24.639616 17044 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:24.730702 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 07:58:24.855955 19134 raft_consensus.cc:1050] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: attempting to promote NON_VOTER 9ad83459fbb549c1a820893b7513cc2e to VOTER
I20240502 07:58:24.856417 19134 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:24.858054 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:24.858284 16977 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:24.858548 19133 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 07:58:24.858733 17044 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:24.858753 19136 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:24.861524 19133 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:24.862109 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:24.863634 16977 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:24.864775 16571 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 5 to 6, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:24.869721 17109 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:24.870950 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:24.871054 16977 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:24.871506 19133 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:24.872146 19133 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:24.873179 19134 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:24.874800 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 904dcca2b5be4bc5a225e201f647c0ea with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:24.874732 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:24.875995 16574 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:24.876582 16977 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
W20240502 07:58:25.029099 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240502 07:58:25.029919 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:25.116693 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:25.143368 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:25.151492 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:25.174973 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240502 07:58:25.175787 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 07:58:25.191583 17109 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:58:25.193292 16977 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20240502 07:58:25.193594 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:25.193766 19091 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:25.195830 19134 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } }
I20240502 07:58:25.196534 16977 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } }
I20240502 07:58:25.198822 16571 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:25.202136 17109 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:25.203600 16977 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 07:58:25.204010 19091 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:25.205662 19134 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
W20240502 07:58:25.205842 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:25.206919 16560 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0ab0feb808e741b0bd50990db02fab68 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:58:25.207805 16977 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:25.207899 16571 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 5 to 6, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:25.209988 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119): Couldn't send request to peer 25ced22802d74d21a8a3dd2ee690463e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ab0feb808e741b0bd50990db02fab68. This is attempt 1: this message will repeat every 5th retry.
W20240502 07:58:25.315367 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:25.368324 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240502 07:58:25.496318 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 904dcca2b5be4bc5a225e201f647c0ea: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:25.606611 19173 ts_tablet_manager.cc:902] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Initiating tablet copy from peer 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:25.607548 19173 tablet_copy_client.cc:308] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:46525
I20240502 07:58:25.611500 17129 tablet_copy_service.cc:140] P 914d87eb1b3345fbbb28738ee49e1455: Received BeginTabletCopySession request for tablet 0ab0feb808e741b0bd50990db02fab68 from peer 25ced22802d74d21a8a3dd2ee690463e ({username='slave'} at 127.16.36.200:33663)
I20240502 07:58:25.611625 17129 tablet_copy_service.cc:161] P 914d87eb1b3345fbbb28738ee49e1455: Beginning new tablet copy session on tablet 0ab0feb808e741b0bd50990db02fab68 from peer 25ced22802d74d21a8a3dd2ee690463e at {username='slave'} at 127.16.36.200:33663: session id = 25ced22802d74d21a8a3dd2ee690463e-0ab0feb808e741b0bd50990db02fab68
I20240502 07:58:25.612403 17129 tablet_copy_source_session.cc:205] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:25.615854 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab0feb808e741b0bd50990db02fab68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:25.623003 19173 tablet_copy_client.cc:783] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:25.623313 19173 tablet_copy_client.cc:647] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:25.625077 19177 tablet_copy_client.cc:1002] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:26.439589 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 07:58:27.074445 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240502 07:58:27.075358 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 904dcca2b5be4bc5a225e201f647c0ea: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:27.125617 19173 tablet_copy_client.cc:523] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 07:58:27.128836 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 07:58:27.188900 19173 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap starting.
I20240502 07:58:27.194857 19173 log.cc:824] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Log is configured to *not* fsync() on all Append() calls
I20240502 07:58:27.217222 19173 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:27.217648 19173 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap complete.
I20240502 07:58:27.218006 19173 ts_tablet_manager.cc:1367] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240502 07:58:27.220156 19173 raft_consensus.cc:348] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:27.220381 19173 raft_consensus.cc:729] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 25ced22802d74d21a8a3dd2ee690463e, State: Initialized, Role: LEARNER
I20240502 07:58:27.220728 19173 consensus_queue.cc:260] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:27.221506 19173 ts_tablet_manager.cc:1398] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 07:58:27.222235 17129 tablet_copy_service.cc:342] P 914d87eb1b3345fbbb28738ee49e1455: Request end of tablet copy session 25ced22802d74d21a8a3dd2ee690463e-0ab0feb808e741b0bd50990db02fab68 received from {username='slave'} at 127.16.36.200:33663
I20240502 07:58:27.222364 17129 tablet_copy_service.cc:433] P 914d87eb1b3345fbbb28738ee49e1455: ending tablet copy session 25ced22802d74d21a8a3dd2ee690463e-0ab0feb808e741b0bd50990db02fab68 on tablet 0ab0feb808e741b0bd50990db02fab68 with peer 25ced22802d74d21a8a3dd2ee690463e
I20240502 07:58:27.291215 18527 raft_consensus.cc:1203] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240502 07:58:27.398468 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:27.451738 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:27.502981 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:27.659031 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:27.661084 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:27.773022 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:27.776031 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 07:58:27.777720 19247 raft_consensus.cc:1050] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: attempting to promote NON_VOTER 25ced22802d74d21a8a3dd2ee690463e to VOTER
I20240502 07:58:27.778193 19247 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:27.779899 18527 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 LEARNER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:58:27.780073 16977 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240502 07:58:27.780447 17044 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:27.780517 19214 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:27.780655 19250 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:27.783453 19250 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:27.784085 18527 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:27.786355 16977 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:27.786546 16565 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 6 to 7, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:27.790272 17109 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:27.791616 18527 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:58:27.791738 16977 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:58:27.792071 19249 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:27.792244 19214 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:27.794188 19247 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:27.794497 16977 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:27.794755 18527 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:27.796371 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ab0feb808e741b0bd50990db02fab68 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:27.797269 16574 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e reported cstate change: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:28.116791 16846 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:58:28.118214 16977 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 07:58:28.118660 19271 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 07:58:28.118898 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:28.120728 19220 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } }
I20240502 07:58:28.121356 16977 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index -1 to 5. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } }
I20240502 07:58:28.123826 16565 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index -1 to 5. New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:28.127307 16846 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:28.129513 16977 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240502 07:58:28.130208 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:28.130889 19271 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:28.132891 19220 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:28.134621 16559 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 54e55a09645e41e69a086b6e13dc8858 with cas_config_opid_index 5: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 07:58:28.134982 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ab0feb808e741b0bd50990db02fab68: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:28.135370 16781 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011): Couldn't send request to peer e9db5c47c0884fca96b3baabba7666cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54e55a09645e41e69a086b6e13dc8858. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:28.135272 16977 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:28.135856 16565 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 5 to 6, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:28.242470 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 07:58:28.736449 19293 ts_tablet_manager.cc:902] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:28.737371 19293 tablet_copy_client.cc:308] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:28.741214 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 54e55a09645e41e69a086b6e13dc8858 from peer e9db5c47c0884fca96b3baabba7666cd ({username='slave'} at 127.16.36.198:39141)
I20240502 07:58:28.741329 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 54e55a09645e41e69a086b6e13dc8858 from peer e9db5c47c0884fca96b3baabba7666cd at {username='slave'} at 127.16.36.198:39141: session id = e9db5c47c0884fca96b3baabba7666cd-54e55a09645e41e69a086b6e13dc8858
I20240502 07:58:28.742224 16867 tablet_copy_source_session.cc:205] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:28.745572 19293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e55a09645e41e69a086b6e13dc8858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:28.752362 19293 tablet_copy_client.cc:783] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:28.752686 19293 tablet_copy_client.cc:647] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:28.758090 19297 tablet_copy_client.cc:1002] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:29.121572 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 07:58:29.151659 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 904dcca2b5be4bc5a225e201f647c0ea: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:30.007117 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 07:58:30.029920 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 07:58:30.078898 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240502 07:58:30.079694 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 07:58:30.187358 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:30.258623 19293 tablet_copy_client.cc:523] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:30.260772 19293 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap starting.
W20240502 07:58:30.262051 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:30.266770 19293 log.cc:824] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Log is configured to *not* fsync() on all Append() calls
W20240502 07:58:30.288133 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:30.288664 19293 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:30.289045 19293 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap complete.
I20240502 07:58:30.289368 19293 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Time spent bootstrapping tablet: real 0.029s user 0.028s sys 0.000s
I20240502 07:58:30.291507 19293 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:30.291749 19293 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e9db5c47c0884fca96b3baabba7666cd, State: Initialized, Role: LEARNER
I20240502 07:58:30.292096 19293 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:30.292977 19293 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:58:30.293814 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session e9db5c47c0884fca96b3baabba7666cd-54e55a09645e41e69a086b6e13dc8858 received from {username='slave'} at 127.16.36.198:39141
I20240502 07:58:30.293900 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session e9db5c47c0884fca96b3baabba7666cd-54e55a09645e41e69a086b6e13dc8858 on tablet 54e55a09645e41e69a086b6e13dc8858 with peer e9db5c47c0884fca96b3baabba7666cd
I20240502 07:58:30.402977 18264 raft_consensus.cc:1203] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
W20240502 07:58:30.426930 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240502 07:58:30.549206 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83decbfaed2a4187bc40c51c35d61747: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:30.660332 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 25 similar messages]
W20240502 07:58:30.670807 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:30.920907 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 07:58:30.939953 19369 raft_consensus.cc:1050] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER e9db5c47c0884fca96b3baabba7666cd to VOTER
I20240502 07:58:30.940384 19369 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:30.941967 16977 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:58:30.942109 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 07:58:30.942608 19368 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:30.943312 19371 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 07:58:30.943420 16782 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:30.946305 19368 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:30.946954 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:30.948783 16977 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:30.949378 16565 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 6 to 7, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:30.953791 16847 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:30.955022 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:58:30.955072 16977 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 07:58:30.955593 19335 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:30.955787 19371 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:30.957265 19335 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:30.958153 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:30.959038 16977 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:30.959627 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54e55a09645e41e69a086b6e13dc8858 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:30.960520 16574 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 7 to 8, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:31.250535 16977 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:58:31.251974 16847 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:31.252490 19391 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:58:31.252677 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:31.254505 19151 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } }
I20240502 07:58:31.254993 16847 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } }
I20240502 07:58:31.257434 16565 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } }
I20240502 07:58:31.261308 16977 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:31.263134 16847 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 07:58:31.264295 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:31.264511 19151 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:31.266655 19393 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
W20240502 07:58:31.267978 16911 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833): Couldn't send request to peer 6c62852374eb46219eba5d56e8564238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ccdfb7bba3b42edae3a322bcd77ab86. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:31.268644 16558 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 07:58:31.268791 16847 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:31.269449 16571 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index 4 to 5, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:31.573376 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 54e55a09645e41e69a086b6e13dc8858: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:31.833246 19412 ts_tablet_manager.cc:902] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: Initiating tablet copy from peer c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:31.833714 19412 tablet_copy_client.cc:308] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:38467
I20240502 07:58:31.837886 16998 tablet_copy_service.cc:140] P c24fcade15af45a79095717955b51756: Received BeginTabletCopySession request for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 from peer 6c62852374eb46219eba5d56e8564238 ({username='slave'} at 127.16.36.197:42105)
I20240502 07:58:31.838093 16998 tablet_copy_service.cc:161] P c24fcade15af45a79095717955b51756: Beginning new tablet copy session on tablet 5ccdfb7bba3b42edae3a322bcd77ab86 from peer 6c62852374eb46219eba5d56e8564238 at {username='slave'} at 127.16.36.197:42105: session id = 6c62852374eb46219eba5d56e8564238-5ccdfb7bba3b42edae3a322bcd77ab86
I20240502 07:58:31.839356 16998 tablet_copy_source_session.cc:205] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:31.840647 19412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccdfb7bba3b42edae3a322bcd77ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:31.842608 19412 tablet_copy_client.cc:783] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:31.842779 19412 tablet_copy_client.cc:647] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:31.844226 19416 tablet_copy_client.cc:1002] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:31.866698 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 07:58:32.094904 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240502 07:58:32.390436 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 07:58:32.472864 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 07:58:32.543665 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 07:58:32.648587 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 07:58:32.836386 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 07:58:33.138298 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 54e55a09645e41e69a086b6e13dc8858: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:33.344627 19412 tablet_copy_client.cc:523] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:33.346030 19412 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: Bootstrap starting.
I20240502 07:58:33.363695 19412 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:33.364006 19412 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: Bootstrap complete.
I20240502 07:58:33.364195 19412 ts_tablet_manager.cc:1367] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240502 07:58:33.364779 19412 raft_consensus.cc:348] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:33.364900 19412 raft_consensus.cc:729] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Initialized, Role: LEARNER
I20240502 07:58:33.365036 19412 consensus_queue.cc:260] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:33.365490 19412 ts_tablet_manager.cc:1398] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 07:58:33.366225 16998 tablet_copy_service.cc:342] P c24fcade15af45a79095717955b51756: Request end of tablet copy session 6c62852374eb46219eba5d56e8564238-5ccdfb7bba3b42edae3a322bcd77ab86 received from {username='slave'} at 127.16.36.197:42105
I20240502 07:58:33.366335 16998 tablet_copy_service.cc:433] P c24fcade15af45a79095717955b51756: ending tablet copy session 6c62852374eb46219eba5d56e8564238-5ccdfb7bba3b42edae3a322bcd77ab86 on tablet 5ccdfb7bba3b42edae3a322bcd77ab86 with peer 6c62852374eb46219eba5d56e8564238
I20240502 07:58:33.473521 18133 raft_consensus.cc:1203] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240502 07:58:33.572794 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 07:58:33.593684 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 07:58:33.994550 19485 raft_consensus.cc:1050] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: attempting to promote NON_VOTER 6c62852374eb46219eba5d56e8564238 to VOTER
I20240502 07:58:33.995028 19485 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:33.996443 18133 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:33.996639 16846 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:33.996966 19488 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:33.997139 19487 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 07:58:33.997498 16913 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:33.999156 19487 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:33.999728 18133 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:34.001497 16846 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:34.002215 16571 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index 5 to 6, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:34.005780 16977 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:34.006942 18133 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:34.007047 16846 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:34.007440 19448 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:34.007596 19485 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:34.009294 19448 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:34.009810 18133 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:34.010023 16846 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:34.012388 16558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:34.012707 16574 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 6 to 7, VOTER 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
W20240502 07:58:34.198940 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 07:58:34.215646 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ccdfb7bba3b42edae3a322bcd77ab86: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:34.427062 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.434131 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.435134 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.435851 16825 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.444427 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.445865 17087 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.448087 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.551393 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.553223 17087 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.555784 18375 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.555891 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:34.564371 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.570876 16825 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:34.587788 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:58:34.597330 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 07:58:34.623811 16977 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:34.625540 16846 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:34.625938 18133 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:34.625959 19587 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:34.626344 19485 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:34.628100 19588 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
W20240502 07:58:34.628787 16911 consensus_peers.cc:485] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 -> Peer 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283): Couldn't send request to peer 9ad83459fbb549c1a820893b7513cc2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5ccdfb7bba3b42edae3a322bcd77ab86. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:34.628693 16846 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:34.631395 18133 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:34.631455 16571 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index 7 to 8, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:34.639487 16846 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:34.641091 16977 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:34.641347 17109 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:34.641700 19600 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 07:58:34.641945 16781 consensus_peers.cc:485] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011): Couldn't send request to peer e9db5c47c0884fca96b3baabba7666cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55aa016d8d254a9e83f94222c92dfff5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:34.641991 19599 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:34.644083 19600 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:34.644562 17109 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:34.644555 16977 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:34.647104 16565 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index -1 to 4, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:34.656191 16846 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:34.657995 17109 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:34.658176 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:34.658421 19600 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:34.658711 19469 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:34.660646 19600 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:34.661710 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:34.661696 17109 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
W20240502 07:58:34.662286 16782 consensus_peers.cc:485] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119): Couldn't send request to peer 25ced22802d74d21a8a3dd2ee690463e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83decbfaed2a4187bc40c51c35d61747. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:34.664216 16569 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 reported cstate change: config changed from index 7 to 8, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:35.025440 19641 ts_tablet_manager.cc:902] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: Initiating tablet copy from peer c24fcade15af45a79095717955b51756 (127.16.36.195:38467)
I20240502 07:58:35.026016 19641 tablet_copy_client.cc:308] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:38467
I20240502 07:58:35.029814 16998 tablet_copy_service.cc:140] P c24fcade15af45a79095717955b51756: Received BeginTabletCopySession request for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 from peer 9ad83459fbb549c1a820893b7513cc2e ({username='slave'} at 127.16.36.199:46511)
I20240502 07:58:35.029948 16998 tablet_copy_service.cc:161] P c24fcade15af45a79095717955b51756: Beginning new tablet copy session on tablet 5ccdfb7bba3b42edae3a322bcd77ab86 from peer 9ad83459fbb549c1a820893b7513cc2e at {username='slave'} at 127.16.36.199:46511: session id = 9ad83459fbb549c1a820893b7513cc2e-5ccdfb7bba3b42edae3a322bcd77ab86
I20240502 07:58:35.030858 16998 tablet_copy_source_session.cc:205] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:35.031730 19641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ccdfb7bba3b42edae3a322bcd77ab86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:35.033663 19641 tablet_copy_client.cc:783] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:35.033812 19641 tablet_copy_client.cc:647] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:35.034893 19645 tablet_copy_client.cc:1002] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Injecting latency into file download: 1500
I20240502 07:58:35.103082 19647 ts_tablet_manager.cc:902] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:35.103596 19647 tablet_copy_client.cc:308] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:35.104166 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 55aa016d8d254a9e83f94222c92dfff5 from peer e9db5c47c0884fca96b3baabba7666cd ({username='slave'} at 127.16.36.198:39141)
I20240502 07:58:35.104293 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 55aa016d8d254a9e83f94222c92dfff5 from peer e9db5c47c0884fca96b3baabba7666cd at {username='slave'} at 127.16.36.198:39141: session id = e9db5c47c0884fca96b3baabba7666cd-55aa016d8d254a9e83f94222c92dfff5
I20240502 07:58:35.105119 16867 tablet_copy_source_session.cc:205] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:35.106058 19647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aa016d8d254a9e83f94222c92dfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:35.107926 19647 tablet_copy_client.cc:783] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:35.108062 19647 tablet_copy_client.cc:647] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:35.109254 19651 tablet_copy_client.cc:1002] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:35.163486 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240502 07:58:35.164217 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 07:58:35.231006 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 54e55a09645e41e69a086b6e13dc8858: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:35.244262 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 07:58:35.262351 19671 ts_tablet_manager.cc:902] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:35.262862 19671 tablet_copy_client.cc:308] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:35.266975 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 83decbfaed2a4187bc40c51c35d61747 from peer 25ced22802d74d21a8a3dd2ee690463e ({username='slave'} at 127.16.36.200:50763)
I20240502 07:58:35.267104 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 83decbfaed2a4187bc40c51c35d61747 from peer 25ced22802d74d21a8a3dd2ee690463e at {username='slave'} at 127.16.36.200:50763: session id = 25ced22802d74d21a8a3dd2ee690463e-83decbfaed2a4187bc40c51c35d61747
I20240502 07:58:35.267908 16867 tablet_copy_source_session.cc:205] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:35.268884 19671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83decbfaed2a4187bc40c51c35d61747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:35.270923 19671 tablet_copy_client.cc:783] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:35.271082 19671 tablet_copy_client.cc:647] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:35.272305 19675 tablet_copy_client.cc:1002] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:35.375315 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 07:58:35.430244 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 07:58:35.554791 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 07:58:35.747844 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 07:58:35.962143 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 07:58:36.243916 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ccdfb7bba3b42edae3a322bcd77ab86: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:36.535269 19641 tablet_copy_client.cc:523] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.536770 19641 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap starting.
I20240502 07:58:36.555039 19641 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.555375 19641 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap complete.
I20240502 07:58:36.555570 19641 ts_tablet_manager.cc:1367] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.001s
I20240502 07:58:36.556180 19641 raft_consensus.cc:348] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:36.556298 19641 raft_consensus.cc:729] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ad83459fbb549c1a820893b7513cc2e, State: Initialized, Role: LEARNER
I20240502 07:58:36.556412 19641 consensus_queue.cc:260] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:36.556754 19641 ts_tablet_manager.cc:1398] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:58:36.557417 16998 tablet_copy_service.cc:342] P c24fcade15af45a79095717955b51756: Request end of tablet copy session 9ad83459fbb549c1a820893b7513cc2e-5ccdfb7bba3b42edae3a322bcd77ab86 received from {username='slave'} at 127.16.36.199:46511
I20240502 07:58:36.557505 16998 tablet_copy_service.cc:433] P c24fcade15af45a79095717955b51756: ending tablet copy session 9ad83459fbb549c1a820893b7513cc2e-5ccdfb7bba3b42edae3a322bcd77ab86 on tablet 5ccdfb7bba3b42edae3a322bcd77ab86 with peer 9ad83459fbb549c1a820893b7513cc2e
I20240502 07:58:36.559795 18395 raft_consensus.cc:1203] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 07:58:36.563757 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 07:58:36.609586 19647 tablet_copy_client.cc:523] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.610970 19647 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap starting.
I20240502 07:58:36.630039 19647 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.630357 19647 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap complete.
I20240502 07:58:36.630548 19647 ts_tablet_manager.cc:1367] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 07:58:36.631112 19647 raft_consensus.cc:348] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:36.631227 19647 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9db5c47c0884fca96b3baabba7666cd, State: Initialized, Role: LEARNER
I20240502 07:58:36.631311 19647 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:36.631794 19647 ts_tablet_manager.cc:1398] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:58:36.632741 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session e9db5c47c0884fca96b3baabba7666cd-55aa016d8d254a9e83f94222c92dfff5 received from {username='slave'} at 127.16.36.198:39141
I20240502 07:58:36.632897 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session e9db5c47c0884fca96b3baabba7666cd-55aa016d8d254a9e83f94222c92dfff5 on tablet 55aa016d8d254a9e83f94222c92dfff5 with peer e9db5c47c0884fca96b3baabba7666cd
I20240502 07:58:36.645584 18264 raft_consensus.cc:1203] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:58:36.772679 19671 tablet_copy_client.cc:523] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:36.774061 19671 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap starting.
I20240502 07:58:36.793236 19671 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:36.793538 19671 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap complete.
I20240502 07:58:36.793709 19671 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 07:58:36.794234 19671 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:36.794348 19671 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25ced22802d74d21a8a3dd2ee690463e, State: Initialized, Role: LEARNER
I20240502 07:58:36.794440 19671 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:36.794744 19671 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:36.795451 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session 25ced22802d74d21a8a3dd2ee690463e-83decbfaed2a4187bc40c51c35d61747 received from {username='slave'} at 127.16.36.200:50763
I20240502 07:58:36.795553 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session 25ced22802d74d21a8a3dd2ee690463e-83decbfaed2a4187bc40c51c35d61747 on tablet 83decbfaed2a4187bc40c51c35d61747 with peer 25ced22802d74d21a8a3dd2ee690463e
I20240502 07:58:36.933930 18527 raft_consensus.cc:1203] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 07:58:37.018361 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 07:58:37.139523 19831 raft_consensus.cc:1050] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756: attempting to promote NON_VOTER 9ad83459fbb549c1a820893b7513cc2e to VOTER
I20240502 07:58:37.140039 19831 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:37.141463 18395 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:37.141546 18133 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:37.141541 16846 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:37.141924 19830 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:37.142132 19831 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:37.142302 19833 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:37.144423 19833 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.144987 18133 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.145171 16846 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.147435 16573 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 reported cstate change: config changed from index 8 to 9, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.148769 18395 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.152403 16977 consensus_queue.cc:237] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:37.153995 18395 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:37.154443 18133 raft_consensus.cc:1261] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer c24fcade15af45a79095717955b51756: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:37.154500 19831 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:37.154887 19838 consensus_queue.cc:1035] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:37.156625 19831 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P c24fcade15af45a79095717955b51756 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.157140 18133 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.157681 18395 raft_consensus.cc:2932] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.158512 16558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.159747 16569 catalog_manager.cc:5484] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 6c62852374eb46219eba5d56e8564238 reported cstate change: config changed from index 9 to 10, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "c24fcade15af45a79095717955b51756" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:37.162432 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ccdfb7bba3b42edae3a322bcd77ab86 with delete_type TABLET_DATA_TOMBSTONED (TS ec22d92eae69437a92d6dc5a1ca43b09 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:48522
I20240502 07:58:37.163028 19844 tablet_replica.cc:330] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: stopping tablet replica
I20240502 07:58:37.163378 19844 raft_consensus.cc:2229] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.163643 19844 raft_consensus.cc:2258] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.165135 19844 ts_tablet_manager.cc:1875] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.168138 19844 ts_tablet_manager.cc:1888] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:58:37.168337 19844 log.cc:1196] T 5ccdfb7bba3b42edae3a322bcd77ab86 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/5ccdfb7bba3b42edae3a322bcd77ab86
I20240502 07:58:37.168970 19709 raft_consensus.cc:1050] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER e9db5c47c0884fca96b3baabba7666cd to VOTER
I20240502 07:58:37.169472 19709 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:37.169528 16559 catalog_manager.cc:4830] TS ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567): tablet 5ccdfb7bba3b42edae3a322bcd77ab86 (table rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]) successfully deleted
I20240502 07:58:37.171159 16977 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:37.171562 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:37.171808 17109 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:37.172371 19812 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:37.172860 19812 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:37.173272 19847 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:37.175290 19846 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.175756 16978 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.177944 17109 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.178007 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.179994 16573 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index 4 to 5, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.183851 16846 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:37.185130 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:37.185173 17109 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:37.185678 19851 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:37.186210 19851 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:37.187296 19846 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.187753 17109 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.188306 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.189285 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55aa016d8d254a9e83f94222c92dfff5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.190624 16571 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:37.193183 16957 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aa016d8d254a9e83f94222c92dfff5 with delete_type TABLET_DATA_TOMBSTONED (TS c24fcade15af45a79095717955b51756 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57588
I20240502 07:58:37.193737 19853 tablet_replica.cc:330] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: stopping tablet replica
I20240502 07:58:37.194062 19853 raft_consensus.cc:2229] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.194332 19853 raft_consensus.cc:2258] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.195381 19853 ts_tablet_manager.cc:1875] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.197428 19853 ts_tablet_manager.cc:1888] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:58:37.197512 19853 log.cc:1196] T 55aa016d8d254a9e83f94222c92dfff5 P c24fcade15af45a79095717955b51756: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/55aa016d8d254a9e83f94222c92dfff5
I20240502 07:58:37.198158 16558 catalog_manager.cc:4830] TS c24fcade15af45a79095717955b51756 (127.16.36.195:38467): tablet 55aa016d8d254a9e83f94222c92dfff5 (table rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]) successfully deleted
I20240502 07:58:37.339251 17109 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:37.341012 16978 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:37.341012 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:37.361284 19877 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:37.361860 19876 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:37.363937 19454 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
W20240502 07:58:37.365301 17042 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833): Couldn't send request to peer 6c62852374eb46219eba5d56e8564238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 904dcca2b5be4bc5a225e201f647c0ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:37.365521 16978 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:37.367146 16565 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 7 to 8, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:37.368357 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:37.376608 17109 consensus_queue.cc:237] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:37.378170 16846 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:37.378396 16978 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:58:37.378401 17042 consensus_peers.cc:485] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283): Couldn't send request to peer 9ad83459fbb549c1a820893b7513cc2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5a8d8cb7f2784b14a64cba5fb33811ab. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:37.378594 19876 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:37.378924 19876 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:37.380569 19876 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:37.381310 16978 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:37.382236 16846 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:37.385274 16569 catalog_manager.cc:5484] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index -1 to 4, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:37.548910 19851 raft_consensus.cc:1050] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER 25ced22802d74d21a8a3dd2ee690463e to VOTER
I20240502 07:58:37.549412 19851 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:37.550930 18527 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:37.550957 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:37.551352 17109 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:37.551358 19709 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:37.551566 19851 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:37.551765 19846 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:37.553548 19851 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.554090 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.554196 17109 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.556650 18527 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.556871 16565 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 8 to 9, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.561196 16846 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:37.562314 18527 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:37.562428 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:37.562836 19846 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:37.563024 19847 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:37.564643 19846 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.565250 18527 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.566190 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83decbfaed2a4187bc40c51c35d61747 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:37.567575 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.568094 16569 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e reported cstate change: config changed from index 9 to 10, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:37.571077 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 83decbfaed2a4187bc40c51c35d61747 with delete_type TABLET_DATA_TOMBSTONED (TS 914d87eb1b3345fbbb28738ee49e1455 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50890
I20240502 07:58:37.571723 19909 tablet_replica.cc:330] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: stopping tablet replica
I20240502 07:58:37.572335 19909 raft_consensus.cc:2229] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:37.573038 19909 raft_consensus.cc:2258] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:37.574317 19909 ts_tablet_manager.cc:1875] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:37.576733 19909 ts_tablet_manager.cc:1888] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:58:37.576829 19909 log.cc:1196] T 83decbfaed2a4187bc40c51c35d61747 P 914d87eb1b3345fbbb28738ee49e1455: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/83decbfaed2a4187bc40c51c35d61747
I20240502 07:58:37.577500 16560 catalog_manager.cc:4830] TS 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525): tablet 83decbfaed2a4187bc40c51c35d61747 (table rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]) successfully deleted
W20240502 07:58:37.587059 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 07:58:37.614336 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 07:58:37.728017 19935 ts_tablet_manager.cc:902] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: Initiating tablet copy from peer 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:37.728627 19935 tablet_copy_client.cc:308] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:46525
W20240502 07:58:37.731786 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 07:58:37.732954 17129 tablet_copy_service.cc:140] P 914d87eb1b3345fbbb28738ee49e1455: Received BeginTabletCopySession request for tablet 904dcca2b5be4bc5a225e201f647c0ea from peer 6c62852374eb46219eba5d56e8564238 ({username='slave'} at 127.16.36.197:47889)
I20240502 07:58:37.733084 17129 tablet_copy_service.cc:161] P 914d87eb1b3345fbbb28738ee49e1455: Beginning new tablet copy session on tablet 904dcca2b5be4bc5a225e201f647c0ea from peer 6c62852374eb46219eba5d56e8564238 at {username='slave'} at 127.16.36.197:47889: session id = 6c62852374eb46219eba5d56e8564238-904dcca2b5be4bc5a225e201f647c0ea
I20240502 07:58:37.733835 17129 tablet_copy_source_session.cc:205] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:37.734798 19935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904dcca2b5be4bc5a225e201f647c0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:37.736671 19935 tablet_copy_client.cc:783] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:37.736853 19935 tablet_copy_client.cc:647] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:37.738204 19939 tablet_copy_client.cc:1002] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: tablet copy: Injecting latency into file download: 1500
I20240502 07:58:37.924535 19951 ts_tablet_manager.cc:902] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: Initiating tablet copy from peer 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:37.925127 19951 tablet_copy_client.cc:308] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:46525
I20240502 07:58:37.925699 17129 tablet_copy_service.cc:140] P 914d87eb1b3345fbbb28738ee49e1455: Received BeginTabletCopySession request for tablet 5a8d8cb7f2784b14a64cba5fb33811ab from peer 9ad83459fbb549c1a820893b7513cc2e ({username='slave'} at 127.16.36.199:52645)
I20240502 07:58:37.925804 17129 tablet_copy_service.cc:161] P 914d87eb1b3345fbbb28738ee49e1455: Beginning new tablet copy session on tablet 5a8d8cb7f2784b14a64cba5fb33811ab from peer 9ad83459fbb549c1a820893b7513cc2e at {username='slave'} at 127.16.36.199:52645: session id = 9ad83459fbb549c1a820893b7513cc2e-5a8d8cb7f2784b14a64cba5fb33811ab
I20240502 07:58:37.926513 17129 tablet_copy_source_session.cc:205] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:37.927413 19951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a8d8cb7f2784b14a64cba5fb33811ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:37.929280 19951 tablet_copy_client.cc:783] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:37.929428 19951 tablet_copy_client.cc:647] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:37.930663 19955 tablet_copy_client.cc:1002] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:38.044924 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 07:58:38.317381 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240502 07:58:38.318387 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ccdfb7bba3b42edae3a322bcd77ab86: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:38.382421 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 07:58:39.207504 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 07:58:39.238590 19935 tablet_copy_client.cc:523] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:39.240108 19935 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: Bootstrap starting.
I20240502 07:58:39.265632 19935 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:39.266003 19935 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: Bootstrap complete.
I20240502 07:58:39.266227 19935 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: Time spent bootstrapping tablet: real 0.026s user 0.024s sys 0.001s
I20240502 07:58:39.266681 19935 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:39.266804 19935 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Initialized, Role: LEARNER
I20240502 07:58:39.266911 19935 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:39.267263 19935 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:39.267997 17129 tablet_copy_service.cc:342] P 914d87eb1b3345fbbb28738ee49e1455: Request end of tablet copy session 6c62852374eb46219eba5d56e8564238-904dcca2b5be4bc5a225e201f647c0ea received from {username='slave'} at 127.16.36.197:47889
I20240502 07:58:39.268112 17129 tablet_copy_service.cc:433] P 914d87eb1b3345fbbb28738ee49e1455: ending tablet copy session 6c62852374eb46219eba5d56e8564238-904dcca2b5be4bc5a225e201f647c0ea on tablet 904dcca2b5be4bc5a225e201f647c0ea with peer 6c62852374eb46219eba5d56e8564238
W20240502 07:58:39.366662 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 54e55a09645e41e69a086b6e13dc8858: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:39.431059 19951 tablet_copy_client.cc:523] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:39.432544 19951 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap starting.
I20240502 07:58:39.450695 19951 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:39.451016 19951 tablet_bootstrap.cc:492] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap complete.
I20240502 07:58:39.451228 19951 ts_tablet_manager.cc:1367] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20240502 07:58:39.451673 19951 raft_consensus.cc:348] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:39.451812 19951 raft_consensus.cc:729] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ad83459fbb549c1a820893b7513cc2e, State: Initialized, Role: LEARNER
I20240502 07:58:39.451922 19951 consensus_queue.cc:260] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:39.452232 19951 ts_tablet_manager.cc:1398] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240502 07:58:39.452762 17129 tablet_copy_service.cc:342] P 914d87eb1b3345fbbb28738ee49e1455: Request end of tablet copy session 9ad83459fbb549c1a820893b7513cc2e-5a8d8cb7f2784b14a64cba5fb33811ab received from {username='slave'} at 127.16.36.199:52645
I20240502 07:58:39.452852 17129 tablet_copy_service.cc:433] P 914d87eb1b3345fbbb28738ee49e1455: ending tablet copy session 9ad83459fbb549c1a820893b7513cc2e-5a8d8cb7f2784b14a64cba5fb33811ab on tablet 5a8d8cb7f2784b14a64cba5fb33811ab with peer 9ad83459fbb549c1a820893b7513cc2e
I20240502 07:58:39.463970 18133 raft_consensus.cc:1203] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
W20240502 07:58:39.570693 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 07:58:39.951751 20062 raft_consensus.cc:1050] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: attempting to promote NON_VOTER 6c62852374eb46219eba5d56e8564238 to VOTER
I20240502 07:58:39.952168 20062 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:39.953640 18133 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:39.953651 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:39.953763 16978 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:39.954131 20062 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:39.954396 20051 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:39.954607 19966 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:39.956508 20051 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.957392 16978 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.958168 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.958690 18133 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.960345 16569 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 8 to 9, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:39.964151 17108 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:39.965377 18133 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:39.965503 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:39.965792 20062 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:39.965987 20051 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:39.967511 20062 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.967964 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.967970 18133 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:39.969637 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 904dcca2b5be4bc5a225e201f647c0ea with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:39.970794 16565 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 9 to 10, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:39.973528 16957 tablet_service.cc:1514] Processing DeleteTablet for tablet 904dcca2b5be4bc5a225e201f647c0ea with delete_type TABLET_DATA_TOMBSTONED (TS c24fcade15af45a79095717955b51756 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:57588
I20240502 07:58:39.973908 20072 tablet_replica.cc:330] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: stopping tablet replica
I20240502 07:58:39.974045 20072 raft_consensus.cc:2229] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:39.974231 20072 raft_consensus.cc:2258] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:39.974817 20072 ts_tablet_manager.cc:1875] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:39.976850 20072 ts_tablet_manager.cc:1888] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 07:58:39.976928 20072 log.cc:1196] T 904dcca2b5be4bc5a225e201f647c0ea P c24fcade15af45a79095717955b51756: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/904dcca2b5be4bc5a225e201f647c0ea
I20240502 07:58:39.977452 16558 catalog_manager.cc:4830] TS c24fcade15af45a79095717955b51756 (127.16.36.195:38467): tablet 904dcca2b5be4bc5a225e201f647c0ea (table rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]) successfully deleted
I20240502 07:58:39.999079 17108 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:39.999872 18395 raft_consensus.cc:1203] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 07:58:40.000903 18527 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:58:40.001338 20051 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:40.001490 16978 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:58:40.001946 20062 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:40.003342 20051 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:40.004799 16978 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:40.005044 18527 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
W20240502 07:58:40.005647 17043 consensus_peers.cc:485] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 -> Peer e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011): Couldn't send request to peer e9db5c47c0884fca96b3baabba7666cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ab0feb808e741b0bd50990db02fab68. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:40.007763 16571 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 8 to 9, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:40.059165 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 07:58:40.168268 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 07:58:40.169178 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 07:58:40.182271 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 07:58:40.515331 20126 ts_tablet_manager.cc:902] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: Initiating tablet copy from peer 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:40.515863 20126 tablet_copy_client.cc:308] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:46525
I20240502 07:58:40.520620 20132 raft_consensus.cc:1050] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455: attempting to promote NON_VOTER 9ad83459fbb549c1a820893b7513cc2e to VOTER
I20240502 07:58:40.520754 17129 tablet_copy_service.cc:140] P 914d87eb1b3345fbbb28738ee49e1455: Received BeginTabletCopySession request for tablet 0ab0feb808e741b0bd50990db02fab68 from peer e9db5c47c0884fca96b3baabba7666cd ({username='slave'} at 127.16.36.198:41113)
I20240502 07:58:40.520850 17129 tablet_copy_service.cc:161] P 914d87eb1b3345fbbb28738ee49e1455: Beginning new tablet copy session on tablet 0ab0feb808e741b0bd50990db02fab68 from peer e9db5c47c0884fca96b3baabba7666cd at {username='slave'} at 127.16.36.198:41113: session id = e9db5c47c0884fca96b3baabba7666cd-0ab0feb808e741b0bd50990db02fab68
I20240502 07:58:40.521119 20132 consensus_queue.cc:237] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:40.521803 17129 tablet_copy_source_session.cc:205] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:40.522786 16846 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:40.522805 16978 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:40.522884 20126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ab0feb808e741b0bd50990db02fab68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:40.522887 18395 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 LEARNER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:40.523352 20133 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:40.523712 20133 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:40.524080 20133 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:40.525400 20126 tablet_copy_client.cc:783] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:40.525584 20126 tablet_copy_client.cc:647] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:40.526042 20133 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.526489 18395 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.526773 20130 tablet_copy_client.cc:1002] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Injecting latency into file download: 1500
I20240502 07:58:40.528786 16978 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.529338 16565 catalog_manager.cc:5484] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 4 to 5, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:40.530539 16846 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.535593 17109 consensus_queue.cc:237] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:40.536808 16978 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:40.536839 18395 raft_consensus.cc:1261] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:40.537298 20135 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:40.537509 19966 consensus_queue.cc:1035] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:40.539332 19966 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.539873 18395 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.539897 16978 raft_consensus.cc:2932] T 5a8d8cb7f2784b14a64cba5fb33811ab P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:40.541083 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5a8d8cb7f2784b14a64cba5fb33811ab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:40.542441 16573 catalog_manager.cc:5484] T 5a8d8cb7f2784b14a64cba5fb33811ab P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 5 to 6, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:40.545231 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a8d8cb7f2784b14a64cba5fb33811ab with delete_type TABLET_DATA_TOMBSTONED (TS ec22d92eae69437a92d6dc5a1ca43b09 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:48522
I20240502 07:58:40.545680 20143 tablet_replica.cc:330] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: stopping tablet replica
I20240502 07:58:40.545822 20143 raft_consensus.cc:2229] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:40.546000 20143 raft_consensus.cc:2258] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:40.546622 20143 ts_tablet_manager.cc:1875] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:40.548625 20143 ts_tablet_manager.cc:1888] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:58:40.548699 20143 log.cc:1196] T 5a8d8cb7f2784b14a64cba5fb33811ab P ec22d92eae69437a92d6dc5a1ca43b09: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/5a8d8cb7f2784b14a64cba5fb33811ab
I20240502 07:58:40.549365 16559 catalog_manager.cc:4830] TS ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567): tablet 5a8d8cb7f2784b14a64cba5fb33811ab (table rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]) successfully deleted
W20240502 07:58:40.600353 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 07:58:40.680508 16846 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:40.682377 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:58:40.682736 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 07:58:40.682847 20161 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:40.683365 20103 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:40.685748 20103 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:40.686393 16978 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
W20240502 07:58:40.686961 16780 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283): Couldn't send request to peer 9ad83459fbb549c1a820893b7513cc2e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54e55a09645e41e69a086b6e13dc8858. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:40.687942 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } } }
I20240502 07:58:40.689147 16571 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 8 to 9, NON_VOTER 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) added. New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:40.749840 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240502 07:58:40.911509 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 07:58:41.058889 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 07:58:41.117743 20188 ts_tablet_manager.cc:902] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:41.118346 20188 tablet_copy_client.cc:308] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:41.122664 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 54e55a09645e41e69a086b6e13dc8858 from peer 9ad83459fbb549c1a820893b7513cc2e ({username='slave'} at 127.16.36.199:50693)
I20240502 07:58:41.122784 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 54e55a09645e41e69a086b6e13dc8858 from peer 9ad83459fbb549c1a820893b7513cc2e at {username='slave'} at 127.16.36.199:50693: session id = 9ad83459fbb549c1a820893b7513cc2e-54e55a09645e41e69a086b6e13dc8858
I20240502 07:58:41.123701 16867 tablet_copy_source_session.cc:205] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:41.124763 20188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e55a09645e41e69a086b6e13dc8858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:41.127386 20188 tablet_copy_client.cc:783] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:41.127573 20188 tablet_copy_client.cc:647] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:41.128777 20192 tablet_copy_client.cc:1002] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:41.479657 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 904dcca2b5be4bc5a225e201f647c0ea: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:41.859398 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 07:58:42.027144 20126 tablet_copy_client.cc:523] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:42.028681 20126 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap starting.
I20240502 07:58:42.048319 20126 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:42.048725 20126 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap complete.
I20240502 07:58:42.048944 20126 ts_tablet_manager.cc:1367] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.004s
I20240502 07:58:42.049434 20126 raft_consensus.cc:348] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:42.049574 20126 raft_consensus.cc:729] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e9db5c47c0884fca96b3baabba7666cd, State: Initialized, Role: LEARNER
I20240502 07:58:42.049677 20126 consensus_queue.cc:260] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:42.050055 20126 ts_tablet_manager.cc:1398] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:58:42.050829 17129 tablet_copy_service.cc:342] P 914d87eb1b3345fbbb28738ee49e1455: Request end of tablet copy session e9db5c47c0884fca96b3baabba7666cd-0ab0feb808e741b0bd50990db02fab68 received from {username='slave'} at 127.16.36.198:41113
I20240502 07:58:42.050947 17129 tablet_copy_service.cc:433] P 914d87eb1b3345fbbb28738ee49e1455: ending tablet copy session e9db5c47c0884fca96b3baabba7666cd-0ab0feb808e741b0bd50990db02fab68 on tablet 0ab0feb808e741b0bd50990db02fab68 with peer e9db5c47c0884fca96b3baabba7666cd
W20240502 07:58:42.096886 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 07:58:42.359612 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 07:58:42.412511 18264 raft_consensus.cc:1203] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 07:58:42.463670 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 07:58:42.589279 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 07:58:42.629163 20188 tablet_copy_client.cc:523] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:42.630920 20188 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap starting.
I20240502 07:58:42.650888 20188 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:42.651211 20188 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Bootstrap complete.
I20240502 07:58:42.651372 20188 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240502 07:58:42.651790 20188 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:42.651909 20188 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 9ad83459fbb549c1a820893b7513cc2e, State: Initialized, Role: LEARNER
I20240502 07:58:42.651986 20188 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: NON_VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: true } }
I20240502 07:58:42.652293 20188 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:58:42.653015 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session 9ad83459fbb549c1a820893b7513cc2e-54e55a09645e41e69a086b6e13dc8858 received from {username='slave'} at 127.16.36.199:50693
I20240502 07:58:42.653126 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session 9ad83459fbb549c1a820893b7513cc2e-54e55a09645e41e69a086b6e13dc8858 on tablet 54e55a09645e41e69a086b6e13dc8858 with peer 9ad83459fbb549c1a820893b7513cc2e
I20240502 07:58:42.784556 18395 raft_consensus.cc:1203] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20240502 07:58:42.884285 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 07:58:42.984635 20300 raft_consensus.cc:1050] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455: attempting to promote NON_VOTER e9db5c47c0884fca96b3baabba7666cd to VOTER
I20240502 07:58:42.985109 20300 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:42.986706 18264 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 LEARNER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:42.986725 16978 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:42.987139 18527 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:42.987115 20299 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:42.987598 20299 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:42.987804 20300 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:42.990094 20302 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:42.990520 18527 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:42.990603 18264 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:42.993265 16565 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e reported cstate change: config changed from index 9 to 10, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:42.994458 16978 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:42.998097 17109 consensus_queue.cc:237] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:42.999349 18527 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:58:42.999372 18264 raft_consensus.cc:1261] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 07:58:42.999790 20230 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:43.000013 20302 consensus_queue.cc:1035] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:43.001729 20230 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 914d87eb1b3345fbbb28738ee49e1455 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:43.002310 18264 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:43.003480 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ab0feb808e741b0bd50990db02fab68 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:43.003803 18527 raft_consensus.cc:2932] T 0ab0feb808e741b0bd50990db02fab68 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:43.004895 16573 catalog_manager.cc:5484] T 0ab0feb808e741b0bd50990db02fab68 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 10 to 11, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:43.007898 16957 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ab0feb808e741b0bd50990db02fab68 with delete_type TABLET_DATA_TOMBSTONED (TS c24fcade15af45a79095717955b51756 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:57588
I20240502 07:58:43.008343 20312 tablet_replica.cc:330] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: stopping tablet replica
I20240502 07:58:43.008541 20312 raft_consensus.cc:2229] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:43.008773 20312 raft_consensus.cc:2258] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:43.009565 20312 ts_tablet_manager.cc:1875] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:43.012012 20312 ts_tablet_manager.cc:1888] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 07:58:43.012099 20312 log.cc:1196] T 0ab0feb808e741b0bd50990db02fab68 P c24fcade15af45a79095717955b51756: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/0ab0feb808e741b0bd50990db02fab68
I20240502 07:58:43.012707 16558 catalog_manager.cc:4830] TS c24fcade15af45a79095717955b51756 (127.16.36.195:38467): tablet 0ab0feb808e741b0bd50990db02fab68 (table rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]) successfully deleted
I20240502 07:58:43.066432 16846 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:43.067893 16978 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 07:58:43.068185 16782 consensus_peers.cc:485] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119): Couldn't send request to peer 25ced22802d74d21a8a3dd2ee690463e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e46524c6180a487a80e3a740844cccdd. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:43.068289 17109 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 07:58:43.068423 20329 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:43.068681 20330 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 07:58:43.070715 20329 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:43.071174 16978 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:43.071246 17109 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:43.074539 16571 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 reported cstate change: config changed from index -1 to 4, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:43.167819 20330 raft_consensus.cc:1050] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER 9ad83459fbb549c1a820893b7513cc2e to VOTER
I20240502 07:58:43.168362 20330 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:43.170475 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:43.170600 18395 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:43.170996 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 07:58:43.171083 20346 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:43.171336 20218 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:43.171484 20329 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:43.173653 20330 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:43.174146 18395 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:43.175277 16978 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:43.175901 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:43.178196 16571 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 9 to 10, 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:43.306267 16846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858"
dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38978
I20240502 07:58:43.306480 16846 raft_consensus.cc:595] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Received request to transfer leadership
W20240502 07:58:43.418260 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 07:58:43.525885 16846 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858"
dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39002
I20240502 07:58:43.526029 16846 raft_consensus.cc:595] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Received request to transfer leadership
I20240502 07:58:43.568024 20218 raft_consensus.cc:979] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: : Instructing follower 9ad83459fbb549c1a820893b7513cc2e to start an election
I20240502 07:58:43.568161 20218 raft_consensus.cc:1067] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Signalling peer 9ad83459fbb549c1a820893b7513cc2e to start an election
I20240502 07:58:43.568881 18395 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858"
dest_uuid: "9ad83459fbb549c1a820893b7513cc2e"
from {username='slave'} at 127.16.36.194:51075
I20240502 07:58:43.569089 18395 raft_consensus.cc:482] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:43.569335 18395 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:58:43.570336 18395 raft_consensus.cc:504] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:43.571547 18395 leader_election.cc:288] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [CANDIDATE]: Term 3 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), c24fcade15af45a79095717955b51756 (127.16.36.195:38467), e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011)
I20240502 07:58:43.571956 16978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9ad83459fbb549c1a820893b7513cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "c24fcade15af45a79095717955b51756"
I20240502 07:58:43.572038 16846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9ad83459fbb549c1a820893b7513cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:43.572108 16978 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:58:43.572172 16846 raft_consensus.cc:3032] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:58:43.572232 16846 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:43.572422 16846 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:43.572662 16846 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:58:43.573220 16978 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad83459fbb549c1a820893b7513cc2e in term 3.
I20240502 07:58:43.573720 16846 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad83459fbb549c1a820893b7513cc2e in term 3.
I20240502 07:58:43.574074 18329 leader_election.cc:302] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9ad83459fbb549c1a820893b7513cc2e, c24fcade15af45a79095717955b51756, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:43.574318 20288 raft_consensus.cc:2781] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:58:43.574510 20288 raft_consensus.cc:686] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 LEADER]: Becoming Leader. State: Replica: 9ad83459fbb549c1a820893b7513cc2e, State: Running, Role: LEADER
I20240502 07:58:43.574834 20288 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:43.576437 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54e55a09645e41e69a086b6e13dc8858" candidate_uuid: "9ad83459fbb549c1a820893b7513cc2e" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: true dest_uuid: "e9db5c47c0884fca96b3baabba7666cd"
I20240502 07:58:43.576663 18264 raft_consensus.cc:3037] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:58:43.577824 18264 raft_consensus.cc:2454] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ad83459fbb549c1a820893b7513cc2e in term 3.
I20240502 07:58:43.577767 16569 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e reported cstate change: term changed from 2 to 3, leader changed from ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) to 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199). New cstate: current_term: 3 leader_uuid: "9ad83459fbb549c1a820893b7513cc2e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:43.584216 20383 ts_tablet_manager.cc:902] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:43.584826 20383 tablet_copy_client.cc:308] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:43.585386 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet e46524c6180a487a80e3a740844cccdd from peer 25ced22802d74d21a8a3dd2ee690463e ({username='slave'} at 127.16.36.200:50763)
I20240502 07:58:43.585492 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet e46524c6180a487a80e3a740844cccdd from peer 25ced22802d74d21a8a3dd2ee690463e at {username='slave'} at 127.16.36.200:50763: session id = 25ced22802d74d21a8a3dd2ee690463e-e46524c6180a487a80e3a740844cccdd
I20240502 07:58:43.586272 16867 tablet_copy_source_session.cc:205] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:43.587210 20383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46524c6180a487a80e3a740844cccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:43.589033 20383 tablet_copy_client.cc:783] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:43.589197 20383 tablet_copy_client.cc:647] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:43.590476 20387 tablet_copy_client.cc:1002] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Injecting latency into file download: 1500
I20240502 07:58:43.972276 16846 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 07:58:43.972872 20288 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:44.064280 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 07:58:44.064829 20288 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:44.068401 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 07:58:44.068845 20288 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 07:58:44.196791 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 07:58:44.434418 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240502 07:58:44.435458 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ab0feb808e741b0bd50990db02fab68: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
I20240502 07:58:44.481396 18395 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } }
I20240502 07:58:44.483024 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 07:58:44.483521 20382 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:44.484443 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 07:58:44.485369 20288 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:44.485689 20414 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:44.486449 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:44.488085 16558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54e55a09645e41e69a086b6e13dc8858 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:44.489084 16573 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e reported cstate change: config changed from index 10 to 12, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New cstate: current_term: 3 leader_uuid: "9ad83459fbb549c1a820893b7513cc2e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:44.489297 16978 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 10 to 12, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } }
I20240502 07:58:44.494153 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e55a09645e41e69a086b6e13dc8858 with delete_type TABLET_DATA_TOMBSTONED (TS ec22d92eae69437a92d6dc5a1ca43b09 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:48522
I20240502 07:58:44.494540 20446 tablet_replica.cc:330] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: stopping tablet replica
I20240502 07:58:44.494707 20446 raft_consensus.cc:2229] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:44.494869 20446 raft_consensus.cc:2258] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:44.495600 20446 ts_tablet_manager.cc:1875] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:44.497716 20446 ts_tablet_manager.cc:1888] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.11
I20240502 07:58:44.497802 20446 log.cc:1196] T 54e55a09645e41e69a086b6e13dc8858 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/54e55a09645e41e69a086b6e13dc8858
I20240502 07:58:44.498391 16559 catalog_manager.cc:4830] TS ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567): tablet 54e55a09645e41e69a086b6e13dc8858 (table rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]) successfully deleted
W20240502 07:58:44.789144 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 07:58:44.793021 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 07:58:44.939786 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 07:58:44.981034 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 07:58:45.090831 20383 tablet_copy_client.cc:523] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:45.092312 20383 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap starting.
I20240502 07:58:45.110618 20383 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:45.110913 20383 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap complete.
I20240502 07:58:45.111070 20383 ts_tablet_manager.cc:1367] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.001s
I20240502 07:58:45.111493 20383 raft_consensus.cc:348] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:45.111608 20383 raft_consensus.cc:729] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25ced22802d74d21a8a3dd2ee690463e, State: Initialized, Role: LEARNER
I20240502 07:58:45.111687 20383 consensus_queue.cc:260] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:45.112020 20383 ts_tablet_manager.cc:1398] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:58:45.112777 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session 25ced22802d74d21a8a3dd2ee690463e-e46524c6180a487a80e3a740844cccdd received from {username='slave'} at 127.16.36.200:50763
I20240502 07:58:45.112857 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session 25ced22802d74d21a8a3dd2ee690463e-e46524c6180a487a80e3a740844cccdd on tablet e46524c6180a487a80e3a740844cccdd with peer 25ced22802d74d21a8a3dd2ee690463e
W20240502 07:58:45.296723 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 07:58:45.427471 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 07:58:45.440171 18527 raft_consensus.cc:1203] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 07:58:45.784292 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 07:58:45.803139 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 07:58:45.826853 20498 raft_consensus.cc:1050] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER 25ced22802d74d21a8a3dd2ee690463e to VOTER
I20240502 07:58:45.827342 20498 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:45.828830 18527 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:45.828909 17109 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:45.829051 16978 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 07:58:45.829298 20500 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:45.829499 20412 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:45.829685 20497 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 07:58:45.831569 20500 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.832049 18527 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.832810 16978 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.834821 17109 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.835215 16565 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e reported cstate change: config changed from index 4 to 5, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.840256 16846 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:45.841538 18527 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:45.841537 17109 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 07:58:45.841940 20498 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:45.842149 20412 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 07:58:45.844107 20500 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.844641 18527 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.844741 17109 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.846633 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e46524c6180a487a80e3a740844cccdd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:45.847522 16571 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 5 to 6, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:45.850260 16957 tablet_service.cc:1514] Processing DeleteTablet for tablet e46524c6180a487a80e3a740844cccdd with delete_type TABLET_DATA_TOMBSTONED (TS c24fcade15af45a79095717955b51756 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57588
I20240502 07:58:45.850598 20513 tablet_replica.cc:330] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: stopping tablet replica
I20240502 07:58:45.850806 20513 raft_consensus.cc:2229] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:45.851001 20513 raft_consensus.cc:2258] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:45.851711 20513 ts_tablet_manager.cc:1875] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:45.853889 20513 ts_tablet_manager.cc:1888] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 07:58:45.853981 20513 log.cc:1196] T e46524c6180a487a80e3a740844cccdd P c24fcade15af45a79095717955b51756: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/e46524c6180a487a80e3a740844cccdd
I20240502 07:58:45.854586 16558 catalog_manager.cc:4830] TS c24fcade15af45a79095717955b51756 (127.16.36.195:38467): tablet e46524c6180a487a80e3a740844cccdd (table rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]) successfully deleted
I20240502 07:58:45.920145 18113 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:45.921541 17087 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:45.924224 18507 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:45.931602 16825 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:45.940595 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:45.940622 18375 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:45.950139 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:45.985719 16846 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:45.987190 17109 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:45.987190 18527 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:58:45.987427 16781 consensus_peers.cc:485] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011): Couldn't send request to peer e9db5c47c0884fca96b3baabba7666cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e46524c6180a487a80e3a740844cccdd. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:45.987679 20412 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:45.987885 20498 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:45.989812 20500 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:45.991748 17109 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:45.991748 18527 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:45.994228 16574 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 6 to 7, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:46.003309 18395 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:46.004897 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 07:58:46.005311 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 07:58:46.005426 20382 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:58:46.005996 20568 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:58:46.007488 20382 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:46.008641 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
W20240502 07:58:46.009115 18328 consensus_peers.cc:485] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e -> Peer 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833): Couldn't send request to peer 6c62852374eb46219eba5d56e8564238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54e55a09645e41e69a086b6e13dc8858. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:46.009629 16978 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:46.010568 16565 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e reported cstate change: config changed from index 12 to 13, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New cstate: current_term: 3 leader_uuid: "9ad83459fbb549c1a820893b7513cc2e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:46.437944 20593 ts_tablet_manager.cc:902] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: Initiating tablet copy from peer 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283)
I20240502 07:58:46.438534 20593 tablet_copy_client.cc:308] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.199:45283
I20240502 07:58:46.443639 18415 tablet_copy_service.cc:140] P 9ad83459fbb549c1a820893b7513cc2e: Received BeginTabletCopySession request for tablet 54e55a09645e41e69a086b6e13dc8858 from peer 6c62852374eb46219eba5d56e8564238 ({username='slave'} at 127.16.36.197:38479)
I20240502 07:58:46.443806 18415 tablet_copy_service.cc:161] P 9ad83459fbb549c1a820893b7513cc2e: Beginning new tablet copy session on tablet 54e55a09645e41e69a086b6e13dc8858 from peer 6c62852374eb46219eba5d56e8564238 at {username='slave'} at 127.16.36.197:38479: session id = 6c62852374eb46219eba5d56e8564238-54e55a09645e41e69a086b6e13dc8858
I20240502 07:58:46.445380 18415 tablet_copy_source_session.cc:205] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:46.446731 20593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e55a09645e41e69a086b6e13dc8858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:46.448747 20593 tablet_copy_client.cc:783] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:46.448930 20593 tablet_copy_client.cc:647] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:46.450488 20597 tablet_copy_client.cc:1002] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:46.566324 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 07:58:46.582557 20613 ts_tablet_manager.cc:902] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:46.583072 20613 tablet_copy_client.cc:308] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:46.583762 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet e46524c6180a487a80e3a740844cccdd from peer e9db5c47c0884fca96b3baabba7666cd ({username='slave'} at 127.16.36.198:39141)
I20240502 07:58:46.583879 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet e46524c6180a487a80e3a740844cccdd from peer e9db5c47c0884fca96b3baabba7666cd at {username='slave'} at 127.16.36.198:39141: session id = e9db5c47c0884fca96b3baabba7666cd-e46524c6180a487a80e3a740844cccdd
I20240502 07:58:46.584803 16867 tablet_copy_source_session.cc:205] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:46.586014 20613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e46524c6180a487a80e3a740844cccdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:46.587951 20613 tablet_copy_client.cc:783] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:46.588109 20613 tablet_copy_client.cc:647] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:46.589288 20617 tablet_copy_client.cc:1002] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:46.747093 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 07:58:47.116297 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 07:58:47.159029 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 07:58:47.339226 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 07:58:47.573412 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 54e55a09645e41e69a086b6e13dc8858: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:47.627012 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 07:58:47.913468 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 07:58:47.950873 20593 tablet_copy_client.cc:523] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:47.952370 20593 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: Bootstrap starting.
I20240502 07:58:47.972935 20593 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:47.973242 20593 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: Bootstrap complete.
I20240502 07:58:47.973434 20593 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.004s
I20240502 07:58:47.973858 20593 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:47.973973 20593 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Initialized, Role: LEARNER
I20240502 07:58:47.974081 20593 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 3.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:47.974404 20593 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:58:47.975148 18415 tablet_copy_service.cc:342] P 9ad83459fbb549c1a820893b7513cc2e: Request end of tablet copy session 6c62852374eb46219eba5d56e8564238-54e55a09645e41e69a086b6e13dc8858 received from {username='slave'} at 127.16.36.197:38479
I20240502 07:58:47.975275 18415 tablet_copy_service.cc:433] P 9ad83459fbb549c1a820893b7513cc2e: ending tablet copy session 6c62852374eb46219eba5d56e8564238-54e55a09645e41e69a086b6e13dc8858 on tablet 54e55a09645e41e69a086b6e13dc8858 with peer 6c62852374eb46219eba5d56e8564238
I20240502 07:58:48.089704 20613 tablet_copy_client.cc:523] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:48.091300 20613 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: Bootstrap starting.
I20240502 07:58:48.107722 18133 raft_consensus.cc:1203] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.12->[3.13-3.13] Dedup: 3.13->[]
I20240502 07:58:48.111768 20613 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:48.112033 20613 tablet_bootstrap.cc:492] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: Bootstrap complete.
I20240502 07:58:48.112191 20613 ts_tablet_manager.cc:1367] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.000s
I20240502 07:58:48.112646 20613 raft_consensus.cc:348] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:48.112763 20613 raft_consensus.cc:729] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9db5c47c0884fca96b3baabba7666cd, State: Initialized, Role: LEARNER
I20240502 07:58:48.112835 20613 consensus_queue.cc:260] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:48.113144 20613 ts_tablet_manager.cc:1398] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd: Time spent starting tablet: real 0.001s user 0.005s sys 0.000s
I20240502 07:58:48.113698 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session e9db5c47c0884fca96b3baabba7666cd-e46524c6180a487a80e3a740844cccdd received from {username='slave'} at 127.16.36.198:39141
I20240502 07:58:48.113797 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session e9db5c47c0884fca96b3baabba7666cd-e46524c6180a487a80e3a740844cccdd on tablet e46524c6180a487a80e3a740844cccdd with peer e9db5c47c0884fca96b3baabba7666cd
W20240502 07:58:48.301563 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 07:58:48.619313 18264 raft_consensus.cc:1203] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 07:58:48.668901 20738 raft_consensus.cc:1050] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e: attempting to promote NON_VOTER 6c62852374eb46219eba5d56e8564238 to VOTER
I20240502 07:58:48.669369 20738 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 10, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:48.670822 18132 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 LEARNER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 07:58:48.670821 16978 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 07:58:48.670835 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240502 07:58:48.671414 20740 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:48.671770 20674 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:48.671937 20738 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:48.674297 20738 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.674677 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.674886 16978 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.677474 16574 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 13 to 14, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "9ad83459fbb549c1a820893b7513cc2e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.681190 18395 consensus_queue.cc:237] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:48.682415 18264 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 07:58:48.682819 20674 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 07:58:48.683183 18132 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 13 to 14, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.684747 20737 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.684896 18133 raft_consensus.cc:1261] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 FOLLOWER]: Refusing update from remote peer 9ad83459fbb549c1a820893b7513cc2e: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20240502 07:58:48.685158 18264 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.685406 20740 consensus_queue.cc:1035] T 54e55a09645e41e69a086b6e13dc8858 P 9ad83459fbb549c1a820893b7513cc2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240502 07:58:48.686389 18133 raft_consensus.cc:2932] T 54e55a09645e41e69a086b6e13dc8858 P 6c62852374eb46219eba5d56e8564238 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.687811 16558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 54e55a09645e41e69a086b6e13dc8858 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:48.688109 16565 catalog_manager.cc:5484] T 54e55a09645e41e69a086b6e13dc8858 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 14 to 15, VOTER c24fcade15af45a79095717955b51756 (127.16.36.195) evicted. New cstate: current_term: 3 leader_uuid: "9ad83459fbb549c1a820893b7513cc2e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:48.691131 16957 tablet_service.cc:1514] Processing DeleteTablet for tablet 54e55a09645e41e69a086b6e13dc8858 with delete_type TABLET_DATA_TOMBSTONED (TS c24fcade15af45a79095717955b51756 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:57588
I20240502 07:58:48.691663 20749 tablet_replica.cc:330] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: stopping tablet replica
I20240502 07:58:48.691792 20749 raft_consensus.cc:2229] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:48.692024 20749 raft_consensus.cc:2258] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:48.692791 20749 ts_tablet_manager.cc:1875] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:48.694900 20749 ts_tablet_manager.cc:1888] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20240502 07:58:48.694976 20749 log.cc:1196] T 54e55a09645e41e69a086b6e13dc8858 P c24fcade15af45a79095717955b51756: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/54e55a09645e41e69a086b6e13dc8858
I20240502 07:58:48.695590 16558 catalog_manager.cc:4830] TS c24fcade15af45a79095717955b51756 (127.16.36.195:38467): tablet 54e55a09645e41e69a086b6e13dc8858 (table rebalance_test_table_4 [id=c78f945f33aa4615bd4d458ce48d4246]) successfully deleted
W20240502 07:58:49.108723 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 07:58:49.241755 20782 raft_consensus.cc:1050] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER e9db5c47c0884fca96b3baabba7666cd to VOTER
I20240502 07:58:49.242218 20782 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:49.243757 18526 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:49.243824 17109 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:49.243858 18264 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:49.244225 20678 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:49.244426 20784 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:49.244715 20785 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:49.246865 20784 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.247550 17109 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.247922 18526 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.249114 18264 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.250146 16569 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 7 to 8, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:49.253859 16846 consensus_queue.cc:237] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:49.255201 18526 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:49.255594 20785 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:49.255692 18264 raft_consensus.cc:1261] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:49.256137 20785 consensus_queue.cc:1035] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:49.257372 20782 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.257895 18526 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.259244 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e46524c6180a487a80e3a740844cccdd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:49.259907 18264 raft_consensus.cc:2932] T e46524c6180a487a80e3a740844cccdd P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:49.260633 16574 catalog_manager.cc:5484] T e46524c6180a487a80e3a740844cccdd P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:49.263510 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet e46524c6180a487a80e3a740844cccdd with delete_type TABLET_DATA_TOMBSTONED (TS 914d87eb1b3345fbbb28738ee49e1455 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50890
I20240502 07:58:49.263968 20794 tablet_replica.cc:330] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: stopping tablet replica
I20240502 07:58:49.264138 20794 raft_consensus.cc:2229] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:49.264374 20794 raft_consensus.cc:2258] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:49.265424 20794 ts_tablet_manager.cc:1875] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:49.267340 20794 ts_tablet_manager.cc:1888] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:58:49.267410 20794 log.cc:1196] T e46524c6180a487a80e3a740844cccdd P 914d87eb1b3345fbbb28738ee49e1455: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/e46524c6180a487a80e3a740844cccdd
I20240502 07:58:49.267974 16560 catalog_manager.cc:4830] TS 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525): tablet e46524c6180a487a80e3a740844cccdd (table rebalance_test_table_0 [id=404be1b7edb446f886d88f35384e9397]) successfully deleted
I20240502 07:58:49.447685 16846 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
W20240502 07:58:49.449419 16781 consensus_peers.cc:485] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011): Couldn't send request to peer e9db5c47c0884fca96b3baabba7666cd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83decbfaed2a4187bc40c51c35d61747. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:49.449427 18526 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:58:49.449860 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:58:49.449838 20784 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:49.450227 20782 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:49.451920 20784 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:49.452514 18526 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:49.453063 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } } }
I20240502 07:58:49.455181 16565 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 10 to 11, NON_VOTER e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:49.463966 17109 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:49.465764 18133 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:58:49.465749 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 07:58:49.465919 17044 consensus_peers.cc:485] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119): Couldn't send request to peer 25ced22802d74d21a8a3dd2ee690463e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 904dcca2b5be4bc5a225e201f647c0ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:49.466197 20816 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:49.466414 20610 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:49.468429 20771 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:49.468961 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:49.468943 18133 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:49.471458 16574 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 reported cstate change: config changed from index 10 to 11, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:49.503111 20765 consensus_queue.cc:579] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.033s)
W20240502 07:58:49.504657 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 07:58:49.550280 20782 consensus_queue.cc:579] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.019s)
I20240502 07:58:49.615945 20830 consensus_queue.cc:579] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.089s)
W20240502 07:58:49.617624 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 07:58:49.660322 20830 consensus_queue.cc:579] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.138s)
W20240502 07:58:49.661943 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 07:58:49.875828 20841 ts_tablet_manager.cc:902] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:49.876320 20841 tablet_copy_client.cc:308] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:49.876888 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 83decbfaed2a4187bc40c51c35d61747 from peer e9db5c47c0884fca96b3baabba7666cd ({username='slave'} at 127.16.36.198:39141)
I20240502 07:58:49.876998 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 83decbfaed2a4187bc40c51c35d61747 from peer e9db5c47c0884fca96b3baabba7666cd at {username='slave'} at 127.16.36.198:39141: session id = e9db5c47c0884fca96b3baabba7666cd-83decbfaed2a4187bc40c51c35d61747
I20240502 07:58:49.877792 16867 tablet_copy_source_session.cc:205] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:49.878640 20841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83decbfaed2a4187bc40c51c35d61747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:49.880440 20841 tablet_copy_client.cc:783] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:49.880622 20841 tablet_copy_client.cc:647] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:49.881652 20845 tablet_copy_client.cc:1002] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:49.958775 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 07:58:50.021466 20856 ts_tablet_manager.cc:902] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: Initiating tablet copy from peer 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525)
I20240502 07:58:50.021984 20856 tablet_copy_client.cc:308] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:46525
I20240502 07:58:50.022588 17129 tablet_copy_service.cc:140] P 914d87eb1b3345fbbb28738ee49e1455: Received BeginTabletCopySession request for tablet 904dcca2b5be4bc5a225e201f647c0ea from peer 25ced22802d74d21a8a3dd2ee690463e ({username='slave'} at 127.16.36.200:33663)
I20240502 07:58:50.022706 17129 tablet_copy_service.cc:161] P 914d87eb1b3345fbbb28738ee49e1455: Beginning new tablet copy session on tablet 904dcca2b5be4bc5a225e201f647c0ea from peer 25ced22802d74d21a8a3dd2ee690463e at {username='slave'} at 127.16.36.200:33663: session id = 25ced22802d74d21a8a3dd2ee690463e-904dcca2b5be4bc5a225e201f647c0ea
I20240502 07:58:50.023505 17129 tablet_copy_source_session.cc:205] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:50.024426 20856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 904dcca2b5be4bc5a225e201f647c0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:50.026443 20856 tablet_copy_client.cc:783] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:50.026576 20856 tablet_copy_client.cc:647] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:50.028004 20860 tablet_copy_client.cc:1002] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Injecting latency into file download: 1500
I20240502 07:58:50.098034 20830 consensus_queue.cc:579] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.247s)
I20240502 07:58:50.100639 20610 consensus_queue.cc:579] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.293s)
W20240502 07:58:50.352823 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 07:58:50.589471 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 07:58:50.666848 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 07:58:50.667632 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5ccdfb7bba3b42edae3a322bcd77ab86: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:51.012426 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240502 07:58:51.134126 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 07:58:51.265523 20884 consensus_queue.cc:579] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Leader has been unable to successfully communicate with peer 9f88eac12f764ac7a1bb5a1bc9111d7f for more than 30 seconds (30.046s)
I20240502 07:58:51.382073 20841 tablet_copy_client.cc:523] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:51.383586 20841 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap starting.
I20240502 07:58:51.403633 20841 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:51.403993 20841 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: Bootstrap complete.
I20240502 07:58:51.404213 20841 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20240502 07:58:51.404723 20841 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:51.404881 20841 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9db5c47c0884fca96b3baabba7666cd, State: Initialized, Role: LEARNER
I20240502 07:58:51.404976 20841 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: NON_VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: true } }
I20240502 07:58:51.405387 20841 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:58:51.406185 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session e9db5c47c0884fca96b3baabba7666cd-83decbfaed2a4187bc40c51c35d61747 received from {username='slave'} at 127.16.36.198:39141
I20240502 07:58:51.406280 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session e9db5c47c0884fca96b3baabba7666cd-83decbfaed2a4187bc40c51c35d61747 on tablet 83decbfaed2a4187bc40c51c35d61747 with peer e9db5c47c0884fca96b3baabba7666cd
I20240502 07:58:51.417517 18264 raft_consensus.cc:1203] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 07:58:51.528482 20856 tablet_copy_client.cc:523] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:51.530021 20856 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap starting.
I20240502 07:58:51.550114 20856 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:51.550437 20856 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap complete.
I20240502 07:58:51.550622 20856 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.002s
I20240502 07:58:51.551067 20856 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:51.551183 20856 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25ced22802d74d21a8a3dd2ee690463e, State: Initialized, Role: LEARNER
I20240502 07:58:51.551283 20856 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:51.551616 20856 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:58:51.552327 17129 tablet_copy_service.cc:342] P 914d87eb1b3345fbbb28738ee49e1455: Request end of tablet copy session 25ced22802d74d21a8a3dd2ee690463e-904dcca2b5be4bc5a225e201f647c0ea received from {username='slave'} at 127.16.36.200:33663
I20240502 07:58:51.552433 17129 tablet_copy_service.cc:433] P 914d87eb1b3345fbbb28738ee49e1455: ending tablet copy session 25ced22802d74d21a8a3dd2ee690463e-904dcca2b5be4bc5a225e201f647c0ea on tablet 904dcca2b5be4bc5a225e201f647c0ea with peer 25ced22802d74d21a8a3dd2ee690463e
W20240502 07:58:51.818392 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 07:58:51.853266 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:51.860083 18526 raft_consensus.cc:1203] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 07:58:51.881934 20967 raft_consensus.cc:1050] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER e9db5c47c0884fca96b3baabba7666cd to VOTER
I20240502 07:58:51.882366 20967 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:51.883839 18264 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:58:51.883889 18526 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:58:51.884263 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:58:51.884244 20968 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:51.884415 20910 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:51.884660 20970 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:51.886634 20970 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:51.887128 18526 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:51.887128 18264 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:51.889364 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:51.890036 16569 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 11 to 12, e9db5c47c0884fca96b3baabba7666cd (127.16.36.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
W20240502 07:58:51.915915 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:52.060906 16847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747"
dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60992
I20240502 07:58:52.061055 16847 raft_consensus.cc:595] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:52.149739 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 07:58:52.225297 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:52.279692 16847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747"
dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32780
I20240502 07:58:52.279839 16847 raft_consensus.cc:595] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:52.350041 20970 raft_consensus.cc:979] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: : Instructing follower 25ced22802d74d21a8a3dd2ee690463e to start an election
I20240502 07:58:52.350220 20970 raft_consensus.cc:1067] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Signalling peer 25ced22802d74d21a8a3dd2ee690463e to start an election
I20240502 07:58:52.350688 18526 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747"
dest_uuid: "25ced22802d74d21a8a3dd2ee690463e"
from {username='slave'} at 127.16.36.194:43107
I20240502 07:58:52.350903 18526 raft_consensus.cc:482] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:52.350998 18526 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.351855 18526 raft_consensus.cc:504] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:52.353036 18526 leader_election.cc:288] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [CANDIDATE]: Term 2 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833), e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011)
I20240502 07:58:52.353359 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "25ced22802d74d21a8a3dd2ee690463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:58:52.353485 16847 raft_consensus.cc:3032] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:52.353529 16847 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:58:52.353682 16847 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:52.353909 16847 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.354943 16847 raft_consensus.cc:2454] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ced22802d74d21a8a3dd2ee690463e in term 2.
I20240502 07:58:52.357214 18133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "25ced22802d74d21a8a3dd2ee690463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "6c62852374eb46219eba5d56e8564238"
I20240502 07:58:52.357419 18133 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.357403 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83decbfaed2a4187bc40c51c35d61747" candidate_uuid: "25ced22802d74d21a8a3dd2ee690463e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "e9db5c47c0884fca96b3baabba7666cd"
I20240502 07:58:52.357509 18264 raft_consensus.cc:3037] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.358346 18264 raft_consensus.cc:2454] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ced22802d74d21a8a3dd2ee690463e in term 2.
I20240502 07:58:52.358356 18133 raft_consensus.cc:2454] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25ced22802d74d21a8a3dd2ee690463e in term 2.
I20240502 07:58:52.358676 18461 leader_election.cc:302] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 25ced22802d74d21a8a3dd2ee690463e, e9db5c47c0884fca96b3baabba7666cd, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:58:52.358978 20946 raft_consensus.cc:2781] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:52.359262 20946 raft_consensus.cc:686] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 2 LEADER]: Becoming Leader. State: Replica: 25ced22802d74d21a8a3dd2ee690463e, State: Running, Role: LEADER
I20240502 07:58:52.359586 20946 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:52.362134 16573 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e reported cstate change: term changed from 1 to 2, leader changed from ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) to 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200). New cstate: current_term: 2 leader_uuid: "25ced22802d74d21a8a3dd2ee690463e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:52.397858 20884 raft_consensus.cc:1050] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: attempting to promote NON_VOTER 25ced22802d74d21a8a3dd2ee690463e to VOTER
I20240502 07:58:52.398280 20884 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:52.399909 18133 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:58:52.399947 18526 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:58:52.399957 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Refusing update from remote peer 914d87eb1b3345fbbb28738ee49e1455: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 07:58:52.400401 21019 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:52.400633 21020 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:52.400801 21017 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:58:52.402745 21020 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:52.403264 18526 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:52.403246 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:52.405176 18133 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:52.406351 16573 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e reported cstate change: config changed from index 11 to 12, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "914d87eb1b3345fbbb28738ee49e1455" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:52.495290 17109 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea"
dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58948
I20240502 07:58:52.495476 17109 raft_consensus.cc:595] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:58:52.543671 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:52.713454 17109 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea"
dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58970
I20240502 07:58:52.713608 17109 raft_consensus.cc:595] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:58:52.817322 21020 raft_consensus.cc:979] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: : Instructing follower 6c62852374eb46219eba5d56e8564238 to start an election
I20240502 07:58:52.817454 21020 raft_consensus.cc:1067] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Signalling peer 6c62852374eb46219eba5d56e8564238 to start an election
I20240502 07:58:52.818090 18133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea"
dest_uuid: "6c62852374eb46219eba5d56e8564238"
from {username='slave'} at 127.16.36.196:42643
I20240502 07:58:52.818332 18133 raft_consensus.cc:482] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:58:52.818410 18133 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.819244 18133 raft_consensus.cc:504] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:52.820355 18133 leader_election.cc:288] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [CANDIDATE]: Term 2 election: Requested vote from peers 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525), 9ad83459fbb549c1a820893b7513cc2e (127.16.36.199:45283), 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119)
I20240502 07:58:52.820705 17109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "6c62852374eb46219eba5d56e8564238" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "914d87eb1b3345fbbb28738ee49e1455"
I20240502 07:58:52.820850 17109 raft_consensus.cc:3032] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:58:52.820896 17109 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 914d87eb1b3345fbbb28738ee49e1455, State: Running, Role: LEADER
I20240502 07:58:52.821075 17109 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:52.821307 17109 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.821369 18395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "6c62852374eb46219eba5d56e8564238" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "9ad83459fbb549c1a820893b7513cc2e"
I20240502 07:58:52.821540 18395 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.822378 17109 raft_consensus.cc:2454] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c62852374eb46219eba5d56e8564238 in term 2.
I20240502 07:58:52.822458 18395 raft_consensus.cc:2454] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c62852374eb46219eba5d56e8564238 in term 2.
I20240502 07:58:52.822810 18066 leader_election.cc:302] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c62852374eb46219eba5d56e8564238, 914d87eb1b3345fbbb28738ee49e1455, 9ad83459fbb549c1a820893b7513cc2e; no voters:
I20240502 07:58:52.823035 21024 raft_consensus.cc:2781] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:58:52.823275 21024 raft_consensus.cc:686] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 2 LEADER]: Becoming Leader. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Running, Role: LEADER
I20240502 07:58:52.823604 21024 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:52.825481 18526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "904dcca2b5be4bc5a225e201f647c0ea" candidate_uuid: "6c62852374eb46219eba5d56e8564238" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "25ced22802d74d21a8a3dd2ee690463e"
I20240502 07:58:52.825634 18526 raft_consensus.cc:3037] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:58:52.826130 16574 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 reported cstate change: term changed from 1 to 2, leader changed from 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) to 6c62852374eb46219eba5d56e8564238 (127.16.36.197). New cstate: current_term: 2 leader_uuid: "6c62852374eb46219eba5d56e8564238" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:58:52.826494 18526 raft_consensus.cc:2454] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c62852374eb46219eba5d56e8564238 in term 2.
I20240502 07:58:52.847103 18264 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer 25ced22802d74d21a8a3dd2ee690463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:58:52.847657 20946 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:58:52.894243 16847 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Refusing update from remote peer 25ced22802d74d21a8a3dd2ee690463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:58:52.894714 20946 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:58:52.898229 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Refusing update from remote peer 25ced22802d74d21a8a3dd2ee690463e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:58:52.898646 21062 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 07:58:53.003430 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:53.302181 18526 consensus_queue.cc:237] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } }
I20240502 07:58:53.303535 18264 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer 25ced22802d74d21a8a3dd2ee690463e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 07:58:53.303536 18133 raft_consensus.cc:1261] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Refusing update from remote peer 25ced22802d74d21a8a3dd2ee690463e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 07:58:53.303939 21062 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:53.304209 21060 consensus_queue.cc:1035] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:53.306033 20946 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 25ced22802d74d21a8a3dd2ee690463e [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:53.306464 18264 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:53.306579 18133 raft_consensus.cc:2932] T 83decbfaed2a4187bc40c51c35d61747 P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:53.308704 16560 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83decbfaed2a4187bc40c51c35d61747 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:53.308981 16569 catalog_manager.cc:5484] T 83decbfaed2a4187bc40c51c35d61747 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 12 to 14, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "25ced22802d74d21a8a3dd2ee690463e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } }
I20240502 07:58:53.311513 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 83decbfaed2a4187bc40c51c35d61747 with delete_type TABLET_DATA_TOMBSTONED (TS ec22d92eae69437a92d6dc5a1ca43b09 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:48522
I20240502 07:58:53.311893 21091 tablet_replica.cc:330] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: stopping tablet replica
I20240502 07:58:53.312031 21091 raft_consensus.cc:2229] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:53.312142 21091 raft_consensus.cc:2258] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:53.312876 21091 ts_tablet_manager.cc:1875] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:53.314882 21091 ts_tablet_manager.cc:1888] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 07:58:53.314961 21091 log.cc:1196] T 83decbfaed2a4187bc40c51c35d61747 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/83decbfaed2a4187bc40c51c35d61747
I20240502 07:58:53.315536 16559 catalog_manager.cc:4830] TS ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567): tablet 83decbfaed2a4187bc40c51c35d61747 (table rebalance_test_table_2 [id=6976ade2781e484db6e6f57edd5df6e4]) successfully deleted
I20240502 07:58:53.330924 17109 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:58:53.331509 21057 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:58:53.389118 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:58:53.389571 21057 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:58:53.392582 18526 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:58:53.393127 21108 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240502 07:58:53.535348 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:54.012653 18133 consensus_queue.cc:237] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:54.013914 18395 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 07:58:54.013959 18526 raft_consensus.cc:1261] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240502 07:58:54.014364 21129 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:54.014691 21108 consensus_queue.cc:1035] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 07:58:54.016186 21129 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:54.016638 18526 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:54.016870 18395 raft_consensus.cc:2932] T 904dcca2b5be4bc5a225e201f647c0ea P 9ad83459fbb549c1a820893b7513cc2e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:54.019109 16574 catalog_manager.cc:5484] T 904dcca2b5be4bc5a225e201f647c0ea P 6c62852374eb46219eba5d56e8564238 reported cstate change: config changed from index 12 to 14, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New cstate: current_term: 2 leader_uuid: "6c62852374eb46219eba5d56e8564238" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "9ad83459fbb549c1a820893b7513cc2e" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 45283 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:54.019289 16558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 904dcca2b5be4bc5a225e201f647c0ea with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:54.021894 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 904dcca2b5be4bc5a225e201f647c0ea with delete_type TABLET_DATA_TOMBSTONED (TS 914d87eb1b3345fbbb28738ee49e1455 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:50890
I20240502 07:58:54.022269 21135 tablet_replica.cc:330] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: stopping tablet replica
I20240502 07:58:54.022415 21135 raft_consensus.cc:2229] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:54.022527 21135 raft_consensus.cc:2258] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:54.023094 21135 ts_tablet_manager.cc:1875] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:54.025043 21135 ts_tablet_manager.cc:1888] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240502 07:58:54.025110 21135 log.cc:1196] T 904dcca2b5be4bc5a225e201f647c0ea P 914d87eb1b3345fbbb28738ee49e1455: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/904dcca2b5be4bc5a225e201f647c0ea
I20240502 07:58:54.025624 16560 catalog_manager.cc:4830] TS 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525): tablet 904dcca2b5be4bc5a225e201f647c0ea (table rebalance_test_table_1 [id=ad3a767add594135ba3ecdc815cb24c7]) successfully deleted
I20240502 07:58:54.204293 16847 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:54.205988 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 07:58:54.206270 17109 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 07:58:54.206339 16780 consensus_peers.cc:485] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 6c62852374eb46219eba5d56e8564238 (127.16.36.197:39833): Couldn't send request to peer 6c62852374eb46219eba5d56e8564238. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55aa016d8d254a9e83f94222c92dfff5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:54.206434 21149 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:54.206714 20970 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 07:58:54.208559 21148 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:54.209074 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:54.209056 17109 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
I20240502 07:58:54.212175 16569 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd reported cstate change: config changed from index 6 to 7, NON_VOTER 6c62852374eb46219eba5d56e8564238 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } } }
W20240502 07:58:54.396435 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 07:58:54.425115 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 07:58:54.541791 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 07:58:54.660007 21170 ts_tablet_manager.cc:902] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:54.660537 21170 tablet_copy_client.cc:308] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:54.661195 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 55aa016d8d254a9e83f94222c92dfff5 from peer 6c62852374eb46219eba5d56e8564238 ({username='slave'} at 127.16.36.197:35345)
I20240502 07:58:54.661317 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 55aa016d8d254a9e83f94222c92dfff5 from peer 6c62852374eb46219eba5d56e8564238 at {username='slave'} at 127.16.36.197:35345: session id = 6c62852374eb46219eba5d56e8564238-55aa016d8d254a9e83f94222c92dfff5
I20240502 07:58:54.662246 16867 tablet_copy_source_session.cc:205] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:54.663156 21170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aa016d8d254a9e83f94222c92dfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:54.665127 21170 tablet_copy_client.cc:783] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:54.665272 21170 tablet_copy_client.cc:647] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:54.666415 21174 tablet_copy_client.cc:1002] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:54.699483 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 07:58:54.936861 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 07:58:55.172204 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 83decbfaed2a4187bc40c51c35d61747: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:55.422658 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 07:58:55.700963 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 07:58:55.843547 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 07:58:56.166765 21170 tablet_copy_client.cc:523] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:56.168169 21170 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: Bootstrap starting.
I20240502 07:58:56.188160 21170 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:56.188649 21170 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: Bootstrap complete.
I20240502 07:58:56.188868 21170 ts_tablet_manager.cc:1367] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: Time spent bootstrapping tablet: real 0.021s user 0.018s sys 0.003s
I20240502 07:58:56.189301 21170 raft_consensus.cc:348] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:56.189422 21170 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Initialized, Role: LEARNER
I20240502 07:58:56.189522 21170 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:58:56.189800 21170 ts_tablet_manager.cc:1398] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:56.190418 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session 6c62852374eb46219eba5d56e8564238-55aa016d8d254a9e83f94222c92dfff5 received from {username='slave'} at 127.16.36.197:35345
I20240502 07:58:56.190500 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session 6c62852374eb46219eba5d56e8564238-55aa016d8d254a9e83f94222c92dfff5 on tablet 55aa016d8d254a9e83f94222c92dfff5 with peer 6c62852374eb46219eba5d56e8564238
W20240502 07:58:56.269683 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240502 07:58:56.601437 18133 raft_consensus.cc:1203] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240502 07:58:56.663502 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 07:58:56.739977 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 07:58:57.011849 21238 raft_consensus.cc:1050] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER 6c62852374eb46219eba5d56e8564238 to VOTER
I20240502 07:58:57.012295 21238 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:57.014096 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:57.014364 18133 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 07:58:57.014513 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 07:58:57.014601 17109 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 07:58:57.014770 21237 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:57.015129 21186 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:57.015322 21240 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 07:58:57.017364 21240 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.017983 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.020128 18133 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.020290 16565 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 7 to 8, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:57.021605 17109 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6c62852374eb46219eba5d56e8564238 (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.024940 16847 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } }
I20240502 07:58:57.026082 18133 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:57.026221 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 07:58:57.026575 21240 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:57.026799 21186 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 07:58:57.028535 21186 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.029088 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.029441 18133 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } }
I20240502 07:58:57.030457 16559 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55aa016d8d254a9e83f94222c92dfff5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:58:57.031385 16565 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 8 to 9, VOTER 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:58:57.033903 17087 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aa016d8d254a9e83f94222c92dfff5 with delete_type TABLET_DATA_TOMBSTONED (TS 914d87eb1b3345fbbb28738ee49e1455 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50890
I20240502 07:58:57.034286 21249 tablet_replica.cc:330] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: stopping tablet replica
I20240502 07:58:57.034427 21249 raft_consensus.cc:2229] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:58:57.034682 21249 raft_consensus.cc:2258] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:58:57.035436 21249 ts_tablet_manager.cc:1875] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:58:57.037889 21249 ts_tablet_manager.cc:1888] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 07:58:57.037971 21249 log.cc:1196] T 55aa016d8d254a9e83f94222c92dfff5 P 914d87eb1b3345fbbb28738ee49e1455: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/55aa016d8d254a9e83f94222c92dfff5
I20240502 07:58:57.038587 16560 catalog_manager.cc:4830] TS 914d87eb1b3345fbbb28738ee49e1455 (127.16.36.196:46525): tablet 55aa016d8d254a9e83f94222c92dfff5 (table rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]) successfully deleted
W20240502 07:58:57.174214 16782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 07:58:57.174916 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 07:58:57.187750 18507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:57.189424 17087 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:58:57.190105 18113 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:57.191527 18375 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:58:57.198262 16825 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:58:57.211342 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:58:57.222790 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:58:57.253823 16847 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:57.255146 18133 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 07:58:57.255380 16782 consensus_peers.cc:485] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119): Couldn't send request to peer 25ced22802d74d21a8a3dd2ee690463e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 55aa016d8d254a9e83f94222c92dfff5. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:58:57.255414 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 07:58:57.255635 21237 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:57.255817 21238 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 07:58:57.257447 21240 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:57.258049 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:57.258112 18133 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } } }
I20240502 07:58:57.260195 16565 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 reported cstate change: config changed from index 9 to 10, NON_VOTER 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 07:58:57.388732 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 07:58:57.765226 21315 ts_tablet_manager.cc:902] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: Initiating tablet copy from peer ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567)
I20240502 07:58:57.765784 21315 tablet_copy_client.cc:308] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:37567
I20240502 07:58:57.766330 16867 tablet_copy_service.cc:140] P ec22d92eae69437a92d6dc5a1ca43b09: Received BeginTabletCopySession request for tablet 55aa016d8d254a9e83f94222c92dfff5 from peer 25ced22802d74d21a8a3dd2ee690463e ({username='slave'} at 127.16.36.200:50763)
I20240502 07:58:57.766443 16867 tablet_copy_service.cc:161] P ec22d92eae69437a92d6dc5a1ca43b09: Beginning new tablet copy session on tablet 55aa016d8d254a9e83f94222c92dfff5 from peer 25ced22802d74d21a8a3dd2ee690463e at {username='slave'} at 127.16.36.200:50763: session id = 25ced22802d74d21a8a3dd2ee690463e-55aa016d8d254a9e83f94222c92dfff5
I20240502 07:58:57.767160 16867 tablet_copy_source_session.cc:205] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:58:57.768010 21315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55aa016d8d254a9e83f94222c92dfff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:58:57.769835 21315 tablet_copy_client.cc:783] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 0 data blocks...
I20240502 07:58:57.769991 21315 tablet_copy_client.cc:647] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Starting download of 1 WAL segments...
I20240502 07:58:57.770977 21319 tablet_copy_client.cc:1002] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Injecting latency into file download: 1500
W20240502 07:58:57.890697 16560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240502 07:58:57.891453 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 904dcca2b5be4bc5a225e201f647c0ea: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:58:58.191006 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 07:58:58.581444 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 07:58:59.172984 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 07:58:59.271306 21315 tablet_copy_client.cc:523] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:58:59.272754 21315 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap starting.
W20240502 07:58:59.280992 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 07:58:59.292114 21315 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:58:59.292398 21315 tablet_bootstrap.cc:492] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: Bootstrap complete.
I20240502 07:58:59.292621 21315 ts_tablet_manager.cc:1367] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 07:58:59.293042 21315 raft_consensus.cc:348] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:59.293155 21315 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 25ced22802d74d21a8a3dd2ee690463e, State: Initialized, Role: LEARNER
I20240502 07:58:59.293233 21315 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: NON_VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: true } }
I20240502 07:58:59.293501 21315 ts_tablet_manager.cc:1398] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:58:59.294252 16867 tablet_copy_service.cc:342] P ec22d92eae69437a92d6dc5a1ca43b09: Request end of tablet copy session 25ced22802d74d21a8a3dd2ee690463e-55aa016d8d254a9e83f94222c92dfff5 received from {username='slave'} at 127.16.36.200:50763
I20240502 07:58:59.294349 16867 tablet_copy_service.cc:433] P ec22d92eae69437a92d6dc5a1ca43b09: ending tablet copy session 25ced22802d74d21a8a3dd2ee690463e-55aa016d8d254a9e83f94222c92dfff5 on tablet 55aa016d8d254a9e83f94222c92dfff5 with peer 25ced22802d74d21a8a3dd2ee690463e
W20240502 07:58:59.335810 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 07:58:59.388614 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 07:58:59.455996 18526 raft_consensus.cc:1203] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240502 07:58:59.782085 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 07:58:59.892021 21381 raft_consensus.cc:1050] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: attempting to promote NON_VOTER 25ced22802d74d21a8a3dd2ee690463e to VOTER
I20240502 07:58:59.892824 21381 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:58:59.894412 18133 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:58:59.894461 18526 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 1 LEARNER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:58:59.894461 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Refusing update from remote peer ec22d92eae69437a92d6dc5a1ca43b09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 07:58:59.894830 21381 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:59.895023 21383 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:59.895228 21380 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 07:58:59.897269 21380 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:59.897733 18133 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:59.899257 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:59.900295 18526 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:59.901227 16571 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 reported cstate change: config changed from index 10 to 11, 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:58:59.960983 16847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5"
dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33076
I20240502 07:58:59.961133 16847 raft_consensus.cc:595] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:00.175068 16847 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5"
dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33086
I20240502 07:59:00.175285 16847 raft_consensus.cc:595] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:00.312536 21381 raft_consensus.cc:979] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: : Instructing follower 6c62852374eb46219eba5d56e8564238 to start an election
I20240502 07:59:00.312687 21381 raft_consensus.cc:1067] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Signalling peer 6c62852374eb46219eba5d56e8564238 to start an election
I20240502 07:59:00.313243 18133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5"
dest_uuid: "6c62852374eb46219eba5d56e8564238"
from {username='slave'} at 127.16.36.194:59235
I20240502 07:59:00.313392 18133 raft_consensus.cc:482] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:00.313462 18133 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:00.314518 18133 raft_consensus.cc:504] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:59:00.315162 18133 leader_election.cc:288] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [CANDIDATE]: Term 2 election: Requested vote from peers ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567), e9db5c47c0884fca96b3baabba7666cd (127.16.36.198:35011), 25ced22802d74d21a8a3dd2ee690463e (127.16.36.200:43119)
I20240502 07:59:00.315805 18526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "6c62852374eb46219eba5d56e8564238" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "25ced22802d74d21a8a3dd2ee690463e"
I20240502 07:59:00.315963 18526 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:00.316856 18526 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c62852374eb46219eba5d56e8564238 in term 2.
I20240502 07:59:00.317348 16847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "6c62852374eb46219eba5d56e8564238" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "ec22d92eae69437a92d6dc5a1ca43b09"
I20240502 07:59:00.317487 16847 raft_consensus.cc:3032] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:00.317538 16847 raft_consensus.cc:729] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ec22d92eae69437a92d6dc5a1ca43b09, State: Running, Role: LEADER
I20240502 07:59:00.317728 16847 consensus_queue.cc:260] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:59:00.317930 16847 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:00.318713 16847 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c62852374eb46219eba5d56e8564238 in term 2.
I20240502 07:59:00.319110 18067 leader_election.cc:302] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 25ced22802d74d21a8a3dd2ee690463e, 6c62852374eb46219eba5d56e8564238, ec22d92eae69437a92d6dc5a1ca43b09; no voters:
I20240502 07:59:00.319522 21344 raft_consensus.cc:2781] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:00.319739 21344 raft_consensus.cc:686] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 2 LEADER]: Becoming Leader. State: Replica: 6c62852374eb46219eba5d56e8564238, State: Running, Role: LEADER
I20240502 07:59:00.319986 21344 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:59:00.322853 16571 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 reported cstate change: term changed from 1 to 2, leader changed from ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) to 6c62852374eb46219eba5d56e8564238 (127.16.36.197). New cstate: current_term: 2 leader_uuid: "6c62852374eb46219eba5d56e8564238" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 07:59:00.324043 18264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55aa016d8d254a9e83f94222c92dfff5" candidate_uuid: "6c62852374eb46219eba5d56e8564238" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "e9db5c47c0884fca96b3baabba7666cd"
I20240502 07:59:00.324199 18264 raft_consensus.cc:3037] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:00.325187 18264 raft_consensus.cc:2454] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c62852374eb46219eba5d56e8564238 in term 2.
W20240502 07:59:00.432199 17044 consensus_peers.cc:485] T 995d54008b154b088b0d8a49e71c93ca P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 07:59:00.457607 16913 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 07:59:00.718374 16847 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 07:59:00.718901 21417 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:00.780946 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 07:59:00.781384 21417 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 07:59:00.784535 18526 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 07:59:00.785387 21431 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 07:59:00.851287 16560 catalog_manager.cc:4573] TS 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0ab0feb808e741b0bd50990db02fab68: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111)
W20240502 07:59:01.326223 17044 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240502 07:59:01.338032 18133 consensus_queue.cc:237] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } }
I20240502 07:59:01.339318 18526 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:59:01.339321 18264 raft_consensus.cc:1261] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Refusing update from remote peer 6c62852374eb46219eba5d56e8564238: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 07:59:01.339711 21454 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:59:01.339898 21431 consensus_queue.cc:1035] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 07:59:01.341572 21454 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:59:01.341998 18264 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P e9db5c47c0884fca96b3baabba7666cd [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:59:01.343331 16558 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 55aa016d8d254a9e83f94222c92dfff5 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 07:59:01.344141 18526 raft_consensus.cc:2932] T 55aa016d8d254a9e83f94222c92dfff5 P 25ced22802d74d21a8a3dd2ee690463e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } } }
I20240502 07:59:01.344530 16565 catalog_manager.cc:5484] T 55aa016d8d254a9e83f94222c92dfff5 P 6c62852374eb46219eba5d56e8564238 reported cstate change: config changed from index 11 to 13, VOTER ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "6c62852374eb46219eba5d56e8564238" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e9db5c47c0884fca96b3baabba7666cd" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "25ced22802d74d21a8a3dd2ee690463e" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 07:59:01.347674 16825 tablet_service.cc:1514] Processing DeleteTablet for tablet 55aa016d8d254a9e83f94222c92dfff5 with delete_type TABLET_DATA_TOMBSTONED (TS ec22d92eae69437a92d6dc5a1ca43b09 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:48522
I20240502 07:59:01.348266 21460 tablet_replica.cc:330] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: stopping tablet replica
I20240502 07:59:01.348412 21460 raft_consensus.cc:2229] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:01.348567 21460 raft_consensus.cc:2258] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:01.349239 21460 ts_tablet_manager.cc:1875] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:01.351181 21460 ts_tablet_manager.cc:1888] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240502 07:59:01.351243 21460 log.cc:1196] T 55aa016d8d254a9e83f94222c92dfff5 P ec22d92eae69437a92d6dc5a1ca43b09: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/55aa016d8d254a9e83f94222c92dfff5
I20240502 07:59:01.351749 16559 catalog_manager.cc:4830] TS ec22d92eae69437a92d6dc5a1ca43b09 (127.16.36.194:37567): tablet 55aa016d8d254a9e83f94222c92dfff5 (table rebalance_test_table_3 [id=2377805d4ed14f6aa205ea4d56c1a439]) successfully deleted
W20240502 07:59:01.385238 16913 consensus_peers.cc:485] T adb927f30a0e453591069d77d90d3077 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 07:59:01.475114 16913 consensus_peers.cc:485] T 543e8d886e14407f9e2468d4802dd3b3 P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 07:59:01.482486 18507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:01.486708 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:01.488632 18113 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:01.503782 18375 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:01.509039 17087 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:01.512670 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:01.513007 16825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:01.606180 18113 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:01.606465 17087 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:01.608911 18507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:01.616441 18375 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:01.617115 16825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:01.624112 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:01.638100 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:59:01.659915 16782 consensus_peers.cc:485] T c7cdc29160a9482d8108af570a5dd404 P ec22d92eae69437a92d6dc5a1ca43b09 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 07:59:01.670536 17044 consensus_peers.cc:485] T 28c98a00679a446e995689f3cea9e91e P 914d87eb1b3345fbbb28738ee49e1455 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.193:37289: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 07:59:01.671098 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:37289
--local_ip_for_outbound_sockets=127.16.36.193
--tserver_master_addrs=127.16.36.254:45191
--webserver_port=39393
--webserver_interface=127.16.36.193
--builtin_ntp_servers=127.16.36.212:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 07:59:01.794975 21538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:01.795284 21538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:01.795367 21538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:01.798676 21538 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 07:59:01.798765 21538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:01.798858 21538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 07:59:01.802651 21538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:37289
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=39393
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.16.36.254:45191
--never_fsync=true
--heap_profile_path=/tmp/kudu.21538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:01.803591 21538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:01.804777 21538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:01.814450 21546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:01.815021 21543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:01.815959 21538 server_base.cc:1034] running on GCE node
W20240502 07:59:01.816602 21544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:01.817067 21538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:01.817796 21538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:01.818969 21538 hybrid_clock.cc:648] HybridClock initialized: now 1714636741818956 us; error 29 us; skew 500 ppm
I20240502 07:59:01.821167 21538 webserver.cc:460] Webserver started at http://127.16.36.193:39393/ using document root <none> and password file <none>
I20240502 07:59:01.821758 21538 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:01.821835 21538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:01.825628 21538 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 07:59:01.828534 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:01.829658 21538 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:01.829777 21538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f"
format_stamp: "Formatted at 2024-05-02 07:58:09 on dist-test-slave-9sjq"
I20240502 07:59:01.830247 21538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:01.843816 21538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:01.844578 21538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:01.844746 21538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:01.845291 21538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:01.846524 21559 ts_tablet_manager.cc:516] Loading tablet metadata (0/12 complete)
I20240502 07:59:01.856359 21538 ts_tablet_manager.cc:559] Loaded tablet metadata (12 total tablets, 12 live tablets)
I20240502 07:59:01.856446 21538 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.000s
I20240502 07:59:01.856573 21538 ts_tablet_manager.cc:574] Registering tablets (0/12 complete)
I20240502 07:59:01.858856 21559 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:01.862279 21538 ts_tablet_manager.cc:590] Registered 12 tablets
I20240502 07:59:01.862345 21538 ts_tablet_manager.cc:569] Time spent register tablets: real 0.006s user 0.001s sys 0.003s
I20240502 07:59:01.874222 21538 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:37289
I20240502 07:59:01.874246 21666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:37289 every 8 connection(s)
I20240502 07:59:01.875912 21538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.IgnoredTserverGoesDownDuringRebalancingTest.TserverDown.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:01.877560 21559 log.cc:824] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:01.877617 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 21538
I20240502 07:59:01.887445 21667 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45191
I20240502 07:59:01.887741 21667 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:01.888561 21667 heartbeater.cc:511] Master 127.16.36.254:45191 requested a full tablet report, sending...
I20240502 07:59:01.891064 16565 ts_manager.cc:194] Re-registered known tserver with Master: 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289)
I20240502 07:59:01.893829 16565 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:34003
I20240502 07:59:01.904881 21559 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:01.907013 21559 tablet_bootstrap.cc:492] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:01.908417 21559 ts_tablet_manager.cc:1367] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.050s user 0.035s sys 0.009s
I20240502 07:59:01.911257 21559 raft_consensus.cc:348] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:59:01.911727 21559 raft_consensus.cc:729] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:01.912274 21559 consensus_queue.cc:260] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:59:01.913033 21667 heartbeater.cc:503] Master 127.16.36.254:45191 was elected leader, sending a full tablet report...
I20240502 07:59:01.913239 21559 ts_tablet_manager.cc:1398] T 5ccdfb7bba3b42edae3a322bcd77ab86 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:59:01.913846 21559 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:01.940965 21559 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:01.941383 21559 tablet_bootstrap.cc:492] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:01.942293 21559 ts_tablet_manager.cc:1367] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.008s
I20240502 07:59:01.942824 21559 raft_consensus.cc:348] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:01.942932 21559 raft_consensus.cc:729] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:01.943071 21559 consensus_queue.cc:260] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:01.943358 21559 ts_tablet_manager.cc:1398] T 904dcca2b5be4bc5a225e201f647c0ea P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:01.943625 21559 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:01.968586 21559 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:01.968955 21559 tablet_bootstrap.cc:492] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:01.969838 21559 ts_tablet_manager.cc:1367] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.026s user 0.013s sys 0.012s
I20240502 07:59:01.970310 21559 raft_consensus.cc:348] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:01.970418 21559 raft_consensus.cc:729] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:01.970549 21559 consensus_queue.cc:260] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:01.970894 21559 ts_tablet_manager.cc:1398] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:01.971060 21559 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:01.995044 21559 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:01.995411 21559 tablet_bootstrap.cc:492] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:01.996284 21559 ts_tablet_manager.cc:1367] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.004s
I20240502 07:59:01.996749 21559 raft_consensus.cc:348] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:59:01.996862 21559 raft_consensus.cc:729] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:01.997017 21559 consensus_queue.cc:260] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:59:01.997314 21559 ts_tablet_manager.cc:1398] T adb927f30a0e453591069d77d90d3077 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 07:59:01.997462 21559 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:02.021991 21559 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.022380 21559 tablet_bootstrap.cc:492] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.023350 21559 ts_tablet_manager.cc:1367] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.012s
I20240502 07:59:02.023772 21559 raft_consensus.cc:348] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:59:02.023875 21559 raft_consensus.cc:729] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.024025 21559 consensus_queue.cc:260] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:59:02.024305 21559 ts_tablet_manager.cc:1398] T c7cdc29160a9482d8108af570a5dd404 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:02.024438 21559 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
W20240502 07:59:02.027890 16913 consensus_peers.cc:485] T c6e66836a49d42b48d1c9be0090565bb P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240502 07:59:02.057744 21559 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.058130 21559 tablet_bootstrap.cc:492] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.059724 21559 ts_tablet_manager.cc:1367] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.035s user 0.025s sys 0.008s
I20240502 07:59:02.060370 21559 raft_consensus.cc:348] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:59:02.060532 21559 raft_consensus.cc:729] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.060700 21559 consensus_queue.cc:260] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:59:02.060977 21559 ts_tablet_manager.cc:1398] T 995d54008b154b088b0d8a49e71c93ca P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:02.061118 21559 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
W20240502 07:59:02.092238 16913 consensus_peers.cc:485] T d42e7b3acd724bd3b47d1dcb2e763f3d P c24fcade15af45a79095717955b51756 -> Peer 9f88eac12f764ac7a1bb5a1bc9111d7f (127.16.36.193:37289): Couldn't send request to peer 9f88eac12f764ac7a1bb5a1bc9111d7f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 86: this message will repeat every 5th retry.
I20240502 07:59:02.093447 21559 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.093988 21559 tablet_bootstrap.cc:492] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.094962 21559 ts_tablet_manager.cc:1367] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.034s user 0.025s sys 0.007s
I20240502 07:59:02.095477 21559 raft_consensus.cc:348] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:59:02.095615 21559 raft_consensus.cc:729] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.095758 21559 consensus_queue.cc:260] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } }
I20240502 07:59:02.096051 21559 ts_tablet_manager.cc:1398] T 54e55a09645e41e69a086b6e13dc8858 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:02.096222 21559 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:02.113045 18375 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:02.118945 17087 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.119062 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:02.124567 18113 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:02.124653 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.140200 21601 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.161432 18507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:02.166973 21559 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.167564 21559 tablet_bootstrap.cc:492] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.168759 21559 ts_tablet_manager.cc:1367] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.073s user 0.029s sys 0.007s
I20240502 07:59:02.169226 21559 raft_consensus.cc:348] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:59:02.169405 21559 raft_consensus.cc:729] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.169582 21559 consensus_queue.cc:260] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } attrs { replace: true } } peers { permanent_uuid: "6c62852374eb46219eba5d56e8564238" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 39833 } attrs { promote: true } }
I20240502 07:59:02.169828 21559 ts_tablet_manager.cc:1398] T 83decbfaed2a4187bc40c51c35d61747 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:02.169977 21559 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:02.177146 16825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:02.201618 21559 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.202013 21559 tablet_bootstrap.cc:492] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.202973 21559 ts_tablet_manager.cc:1367] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20240502 07:59:02.203413 21559 raft_consensus.cc:348] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:59:02.203512 21559 raft_consensus.cc:729] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.203673 21559 consensus_queue.cc:260] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } }
I20240502 07:59:02.203934 21559 ts_tablet_manager.cc:1398] T d42e7b3acd724bd3b47d1dcb2e763f3d P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 07:59:02.204095 21559 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:02.234112 21559 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.234481 21559 tablet_bootstrap.cc:492] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.235507 21559 ts_tablet_manager.cc:1367] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20240502 07:59:02.235935 21559 raft_consensus.cc:348] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:59:02.236043 21559 raft_consensus.cc:729] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.236176 21559 consensus_queue.cc:260] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "ec22d92eae69437a92d6dc5a1ca43b09" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 37567 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } }
I20240502 07:59:02.236444 21559 ts_tablet_manager.cc:1398] T c6e66836a49d42b48d1c9be0090565bb P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:02.236615 21559 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:02.246896 21612 raft_consensus.cc:3037] T 28c98a00679a446e995689f3cea9e91e P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:02.267822 21559 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.268283 21559 tablet_bootstrap.cc:492] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.269490 21559 ts_tablet_manager.cc:1367] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20240502 07:59:02.270026 21559 raft_consensus.cc:348] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:02.270153 21559 raft_consensus.cc:729] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.270299 21559 consensus_queue.cc:260] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:02.270596 21559 ts_tablet_manager.cc:1398] T 0ab0feb808e741b0bd50990db02fab68 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:02.270787 21559 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap starting.
I20240502 07:59:02.289284 21601 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.293486 17087 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:02.293752 18113 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:02.294039 18507 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:02.308657 21559 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:02.309142 21559 tablet_bootstrap.cc:492] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Bootstrap complete.
I20240502 07:59:02.310339 21559 ts_tablet_manager.cc:1367] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent bootstrapping tablet: real 0.040s user 0.030s sys 0.004s
I20240502 07:59:02.310820 21559 raft_consensus.cc:348] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:02.310952 21559 raft_consensus.cc:729] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f88eac12f764ac7a1bb5a1bc9111d7f, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.311081 21559 consensus_queue.cc:260] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c24fcade15af45a79095717955b51756" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 38467 } } peers { permanent_uuid: "9f88eac12f764ac7a1bb5a1bc9111d7f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 37289 } } peers { permanent_uuid: "914d87eb1b3345fbbb28738ee49e1455" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 46525 } }
I20240502 07:59:02.311367 21559 ts_tablet_manager.cc:1398] T 543e8d886e14407f9e2468d4802dd3b3 P 9f88eac12f764ac7a1bb5a1bc9111d7f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:02.316439 16957 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:02.319463 18244 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:02.320659 18375 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:02.329190 16825 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:02.370534 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 21538
I20240502 07:59:02.379036 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16765
I20240502 07:59:02.385960 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16896
I20240502 07:59:02.393190 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17027
I20240502 07:59:02.400229 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 18051
I20240502 07:59:02.406738 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 18182
I20240502 07:59:02.412775 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 18312
I20240502 07:59:02.419199 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 18443
I20240502 07:59:02.425532 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16543
2024-05-02T07:59:02Z chronyd exiting
[ OK ] IgnoredTserverGoesDownDuringRebalancingTest.TserverDown (53779 ms)
[----------] 1 test from IgnoredTserverGoesDownDuringRebalancingTest (53779 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0
I20240502 07:59:02.462746 16531 test_util.cc:268] Using random seed: -101446381
I20240502 07:59:02.463552 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:59:02.463598 16531 ts_itest-base.cc:113] --------------
I20240502 07:59:02.463608 16531 ts_itest-base.cc:114] 5 tablet servers
I20240502 07:59:02.463618 16531 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:59:02.463631 16531 ts_itest-base.cc:116] --------------
2024-05-02T07:59:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:59:02Z Disabled control of system clock
I20240502 07:59:02.472443 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:33429
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:33429
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 07:59:02.582069 21776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.582340 21776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:02.582381 21776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.585558 21776 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:02.585642 21776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:59:02.585661 21776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:02.585687 21776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:59:02.585726 21776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:59:02.589658 21776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:33429
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:33429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.21776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:02.590644 21776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:02.591807 21776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:02.599824 21781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:02.600644 21784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:02.601265 21782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:02.602181 21776 server_base.cc:1034] running on GCE node
I20240502 07:59:02.602761 21776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:02.603739 21776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:02.604895 21776 hybrid_clock.cc:648] HybridClock initialized: now 1714636742604883 us; error 27 us; skew 500 ppm
I20240502 07:59:02.606878 21776 webserver.cc:460] Webserver started at http://127.16.36.254:43191/ using document root <none> and password file <none>
I20240502 07:59:02.607447 21776 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:02.607509 21776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:02.607683 21776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:02.609417 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a74c726fb79f45d28ddb3db1d0d994da"
format_stamp: "Formatted at 2024-05-02 07:59:02 on dist-test-slave-9sjq"
I20240502 07:59:02.609890 21776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a74c726fb79f45d28ddb3db1d0d994da"
format_stamp: "Formatted at 2024-05-02 07:59:02 on dist-test-slave-9sjq"
I20240502 07:59:02.613236 21776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:02.615474 21790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.616544 21776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:02.616669 21776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a74c726fb79f45d28ddb3db1d0d994da"
format_stamp: "Formatted at 2024-05-02 07:59:02 on dist-test-slave-9sjq"
I20240502 07:59:02.616763 21776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:02.635231 21776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:02.635915 21776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:02.636087 21776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:02.642997 21776 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:33429
I20240502 07:59:02.643028 21842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:33429 every 8 connection(s)
I20240502 07:59:02.643985 21776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:59:02.646711 21843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:02.648757 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 21776
I20240502 07:59:02.648885 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:59:02.652616 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da: Bootstrap starting.
I20240502 07:59:02.654708 21843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:02.655517 21843 log.cc:824] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:02.657395 21843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da: No bootstrap required, opened a new log
I20240502 07:59:02.659974 21843 raft_consensus.cc:348] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74c726fb79f45d28ddb3db1d0d994da" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33429 } }
I20240502 07:59:02.660171 21843 raft_consensus.cc:374] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:02.660250 21843 raft_consensus.cc:729] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a74c726fb79f45d28ddb3db1d0d994da, State: Initialized, Role: FOLLOWER
I20240502 07:59:02.660704 21843 consensus_queue.cc:260] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74c726fb79f45d28ddb3db1d0d994da" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33429 } }
I20240502 07:59:02.660835 21843 raft_consensus.cc:388] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:02.660899 21843 raft_consensus.cc:482] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:02.661010 21843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:02.661774 21843 raft_consensus.cc:504] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74c726fb79f45d28ddb3db1d0d994da" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33429 } }
I20240502 07:59:02.662089 21843 leader_election.cc:302] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a74c726fb79f45d28ddb3db1d0d994da; no voters:
I20240502 07:59:02.662384 21843 leader_election.cc:288] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:02.662536 21848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:02.662746 21848 raft_consensus.cc:686] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [term 1 LEADER]: Becoming Leader. State: Replica: a74c726fb79f45d28ddb3db1d0d994da, State: Running, Role: LEADER
I20240502 07:59:02.663554 21843 sys_catalog.cc:564] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:02.663110 21848 consensus_queue.cc:237] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74c726fb79f45d28ddb3db1d0d994da" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33429 } }
I20240502 07:59:02.665196 21849 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a74c726fb79f45d28ddb3db1d0d994da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74c726fb79f45d28ddb3db1d0d994da" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33429 } } }
I20240502 07:59:02.665196 21850 sys_catalog.cc:455] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [sys.catalog]: SysCatalogTable state changed. Reason: New leader a74c726fb79f45d28ddb3db1d0d994da. Latest consensus state: current_term: 1 leader_uuid: "a74c726fb79f45d28ddb3db1d0d994da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a74c726fb79f45d28ddb3db1d0d994da" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33429 } } }
I20240502 07:59:02.665349 21849 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:02.665374 21850 sys_catalog.cc:458] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da [sys.catalog]: This master's current role is: LEADER
W20240502 07:59:02.668581 21863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:59:02.668649 21863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:59:02.668730 21862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:02.669418 21862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:02.674160 21862 catalog_manager.cc:1347] Generated new cluster ID: ae3776b3bad54c898d93825029c4ec7a
I20240502 07:59:02.674247 21862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:02.684821 21862 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:02.685767 21862 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:02.693641 21862 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a74c726fb79f45d28ddb3db1d0d994da: Generated new TSK 0
I20240502 07:59:02.694361 21862 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:02.704216 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--builtin_ntp_servers=127.16.36.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:02.814663 21867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.814890 21867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:02.814937 21867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.818090 21867 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:02.818166 21867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:02.818245 21867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 07:59:02.822145 21867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--never_fsync=true
--heap_profile_path=/tmp/kudu.21867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:02.823058 21867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:02.824240 21867 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:02.832890 21875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:02.833400 21872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:02.834278 21873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:02.835091 21867 server_base.cc:1034] running on GCE node
I20240502 07:59:02.835544 21867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:02.836109 21867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:02.837253 21867 hybrid_clock.cc:648] HybridClock initialized: now 1714636742837239 us; error 28 us; skew 500 ppm
I20240502 07:59:02.839155 21867 webserver.cc:460] Webserver started at http://127.16.36.193:36695/ using document root <none> and password file <none>
I20240502 07:59:02.839723 21867 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:02.839776 21867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:02.839946 21867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:02.841647 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a5bfecc3551b421d87d2abc1488bc4f6"
format_stamp: "Formatted at 2024-05-02 07:59:02 on dist-test-slave-9sjq"
I20240502 07:59:02.842128 21867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a5bfecc3551b421d87d2abc1488bc4f6"
format_stamp: "Formatted at 2024-05-02 07:59:02 on dist-test-slave-9sjq"
I20240502 07:59:02.845479 21867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 07:59:02.847666 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.848675 21867 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:02.848786 21867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a5bfecc3551b421d87d2abc1488bc4f6"
format_stamp: "Formatted at 2024-05-02 07:59:02 on dist-test-slave-9sjq"
I20240502 07:59:02.848886 21867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:02.867890 21867 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:02.868563 21867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:02.868739 21867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:02.869294 21867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:02.870229 21867 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:02.870285 21867 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.870348 21867 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:02.870375 21867 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:02.878933 21867 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:40903
I20240502 07:59:02.878955 21994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:40903 every 8 connection(s)
I20240502 07:59:02.879884 21867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:02.880353 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 21867
I20240502 07:59:02.880486 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:59:02.882730 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--builtin_ntp_servers=127.16.36.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:02.890405 21995 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33429
I20240502 07:59:02.890676 21995 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:02.891253 21995 heartbeater.cc:511] Master 127.16.36.254:33429 requested a full tablet report, sending...
I20240502 07:59:02.892872 21807 ts_manager.cc:194] Registered new tserver with Master: a5bfecc3551b421d87d2abc1488bc4f6 (127.16.36.193:40903)
I20240502 07:59:02.894076 21807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:42445
W20240502 07:59:02.993364 21998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:02.993618 21998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:02.993705 21998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:02.996827 21998 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:02.996912 21998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:02.997009 21998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 07:59:03.000802 21998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--never_fsync=true
--heap_profile_path=/tmp/kudu.21998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:03.001775 21998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:03.002947 21998 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:03.012290 22006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.012766 22004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.013672 22003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:03.014469 21998 server_base.cc:1034] running on GCE node
I20240502 07:59:03.014919 21998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:03.015544 21998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:03.016724 21998 hybrid_clock.cc:648] HybridClock initialized: now 1714636743016711 us; error 41 us; skew 500 ppm
I20240502 07:59:03.018915 21998 webserver.cc:460] Webserver started at http://127.16.36.194:42227/ using document root <none> and password file <none>
I20240502 07:59:03.019484 21998 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:03.019548 21998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:03.019734 21998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:03.021550 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "138e27ede9534845ad00e872a3c3df1c"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.022063 21998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "138e27ede9534845ad00e872a3c3df1c"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.025496 21998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240502 07:59:03.027732 22012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.028828 21998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:03.028944 21998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "138e27ede9534845ad00e872a3c3df1c"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.029028 21998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:03.053746 21998 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:03.054430 21998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:03.054601 21998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:03.055157 21998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:03.056046 21998 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:03.056105 21998 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.056169 21998 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:03.056195 21998 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.065062 21998 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:37017
I20240502 07:59:03.065089 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:37017 every 8 connection(s)
I20240502 07:59:03.066051 21998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:59:03.069046 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 21998
I20240502 07:59:03.069155 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:59:03.071189 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--builtin_ntp_servers=127.16.36.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:03.075129 22126 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33429
I20240502 07:59:03.075407 22126 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:03.076052 22126 heartbeater.cc:511] Master 127.16.36.254:33429 requested a full tablet report, sending...
I20240502 07:59:03.077226 21807 ts_manager.cc:194] Registered new tserver with Master: 138e27ede9534845ad00e872a3c3df1c (127.16.36.194:37017)
I20240502 07:59:03.077795 21807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:34605
W20240502 07:59:03.184711 22129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:03.184979 22129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:03.185050 22129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:03.188206 22129 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:03.188306 22129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:03.188413 22129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 07:59:03.192214 22129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--never_fsync=true
--heap_profile_path=/tmp/kudu.22129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:03.193270 22129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:03.194466 22129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:03.203974 22134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.204735 22137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.205467 22135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:03.206418 22129 server_base.cc:1034] running on GCE node
I20240502 07:59:03.206912 22129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:03.207552 22129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:03.208714 22129 hybrid_clock.cc:648] HybridClock initialized: now 1714636743208697 us; error 34 us; skew 500 ppm
I20240502 07:59:03.210799 22129 webserver.cc:460] Webserver started at http://127.16.36.195:37079/ using document root <none> and password file <none>
I20240502 07:59:03.211378 22129 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:03.211441 22129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:03.211621 22129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:03.213410 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "223e69f38db94cfc945285235919ba74"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.213891 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "223e69f38db94cfc945285235919ba74"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.217396 22129 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:03.219612 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.220716 22129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:03.220865 22129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "223e69f38db94cfc945285235919ba74"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.220960 22129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:03.241722 22129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:03.242406 22129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:03.242579 22129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:03.243121 22129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:03.244031 22129 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:03.244083 22129 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.244145 22129 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:03.244170 22129 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.253201 22129 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:46531
I20240502 07:59:03.253219 22256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:46531 every 8 connection(s)
I20240502 07:59:03.254182 22129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:03.257692 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 22129
I20240502 07:59:03.257834 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:59:03.260051 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--builtin_ntp_servers=127.16.36.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:03.264185 22257 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33429
I20240502 07:59:03.264503 22257 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:03.265166 22257 heartbeater.cc:511] Master 127.16.36.254:33429 requested a full tablet report, sending...
I20240502 07:59:03.266445 21807 ts_manager.cc:194] Registered new tserver with Master: 223e69f38db94cfc945285235919ba74 (127.16.36.195:46531)
I20240502 07:59:03.267163 21807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:46785
W20240502 07:59:03.371098 22260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:03.371342 22260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:03.371390 22260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:03.374580 22260 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:03.374668 22260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:03.374743 22260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 07:59:03.378546 22260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--never_fsync=true
--heap_profile_path=/tmp/kudu.22260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:03.379523 22260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:03.380700 22260 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:03.389838 22268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.390285 22265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.391249 22266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:03.391976 22260 server_base.cc:1034] running on GCE node
I20240502 07:59:03.392386 22260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:03.392985 22260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:03.394196 22260 hybrid_clock.cc:648] HybridClock initialized: now 1714636743394171 us; error 42 us; skew 500 ppm
I20240502 07:59:03.396243 22260 webserver.cc:460] Webserver started at http://127.16.36.196:38497/ using document root <none> and password file <none>
I20240502 07:59:03.396833 22260 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:03.396896 22260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:03.397071 22260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:03.398789 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e8afad6cf7ec44d88ff95b09ec01d077"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.399255 22260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e8afad6cf7ec44d88ff95b09ec01d077"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.402863 22260 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:03.405242 22274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.406399 22260 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:03.406538 22260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e8afad6cf7ec44d88ff95b09ec01d077"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.406634 22260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:03.440670 22260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:03.441326 22260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:03.441501 22260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:03.442049 22260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:03.442970 22260 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:03.443024 22260 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.443089 22260 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:03.443118 22260 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.451570 22260 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:33515
I20240502 07:59:03.451589 22387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:33515 every 8 connection(s)
I20240502 07:59:03.452642 22260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:03.456779 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 22260
I20240502 07:59:03.456913 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:59:03.459086 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:0
--local_ip_for_outbound_sockets=127.16.36.197
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--builtin_ntp_servers=127.16.36.212:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:03.462352 22388 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33429
I20240502 07:59:03.462637 22388 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:03.463351 22388 heartbeater.cc:511] Master 127.16.36.254:33429 requested a full tablet report, sending...
I20240502 07:59:03.464673 21807 ts_manager.cc:194] Registered new tserver with Master: e8afad6cf7ec44d88ff95b09ec01d077 (127.16.36.196:33515)
I20240502 07:59:03.465332 21807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:48403
W20240502 07:59:03.572679 22391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:03.572926 22391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:03.572971 22391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:03.576035 22391 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:03.576104 22391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:03.576203 22391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:59:03.579876 22391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33429
--never_fsync=true
--heap_profile_path=/tmp/kudu.22391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:03.580842 22391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:03.581967 22391 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:03.590916 22397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.591681 22396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:03.592324 22399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:03.593210 22391 server_base.cc:1034] running on GCE node
I20240502 07:59:03.593627 22391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:03.594264 22391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:03.595433 22391 hybrid_clock.cc:648] HybridClock initialized: now 1714636743595406 us; error 41 us; skew 500 ppm
I20240502 07:59:03.597533 22391 webserver.cc:460] Webserver started at http://127.16.36.197:41349/ using document root <none> and password file <none>
I20240502 07:59:03.598119 22391 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:03.598181 22391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:03.598362 22391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:03.600091 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "8ae13fc4213741f9a0d9ffde9f01b0f8"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.600651 22391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "8ae13fc4213741f9a0d9ffde9f01b0f8"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.604092 22391 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:03.606411 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.607406 22391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:03.607517 22391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "8ae13fc4213741f9a0d9ffde9f01b0f8"
format_stamp: "Formatted at 2024-05-02 07:59:03 on dist-test-slave-9sjq"
I20240502 07:59:03.607607 22391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:03.628095 22391 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:03.628818 22391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:03.628988 22391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:03.629539 22391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:03.630502 22391 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:03.630555 22391 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.630613 22391 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:03.630641 22391 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:03.639204 22391 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:43885
I20240502 07:59:03.639222 22518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:43885 every 8 connection(s)
I20240502 07:59:03.640192 22391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:03.645480 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 22391
I20240502 07:59:03.645601 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceStartCriteriaTest.UnknownIgnoredTServer_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:59:03.649956 22519 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33429
I20240502 07:59:03.650206 22519 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:03.650676 22519 heartbeater.cc:511] Master 127.16.36.254:33429 requested a full tablet report, sending...
I20240502 07:59:03.651561 21807 ts_manager.cc:194] Registered new tserver with Master: 8ae13fc4213741f9a0d9ffde9f01b0f8 (127.16.36.197:43885)
I20240502 07:59:03.652094 21807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:42133
I20240502 07:59:03.659798 16531 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 07:59:03.668917 21807 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45186:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 07:59:03.686678 22321 tablet_service.cc:1467] Processing CreateTablet for tablet 99e0220e4d3b4089b30c1b010652cd69 (DEFAULT_TABLE table=TestTable [id=5d9cffef2b8c49ddacf62c177a04c5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:03.686678 22191 tablet_service.cc:1467] Processing CreateTablet for tablet 99e0220e4d3b4089b30c1b010652cd69 (DEFAULT_TABLE table=TestTable [id=5d9cffef2b8c49ddacf62c177a04c5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:03.686681 21929 tablet_service.cc:1467] Processing CreateTablet for tablet 99e0220e4d3b4089b30c1b010652cd69 (DEFAULT_TABLE table=TestTable [id=5d9cffef2b8c49ddacf62c177a04c5c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:03.687595 21929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e0220e4d3b4089b30c1b010652cd69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:03.687602 22191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e0220e4d3b4089b30c1b010652cd69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:03.687602 22321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e0220e4d3b4089b30c1b010652cd69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:03.693440 22536 tablet_bootstrap.cc:492] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: Bootstrap starting.
I20240502 07:59:03.693935 22538 tablet_bootstrap.cc:492] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: Bootstrap starting.
I20240502 07:59:03.694042 22537 tablet_bootstrap.cc:492] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: Bootstrap starting.
I20240502 07:59:03.695407 22536 tablet_bootstrap.cc:654] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:03.695940 22538 tablet_bootstrap.cc:654] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:03.695940 22537 tablet_bootstrap.cc:654] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:03.696105 22536 log.cc:824] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:03.696748 22537 log.cc:824] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:03.696738 22538 log.cc:824] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:03.697898 22536 tablet_bootstrap.cc:492] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: No bootstrap required, opened a new log
I20240502 07:59:03.698073 22536 ts_tablet_manager.cc:1367] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 07:59:03.698383 22537 tablet_bootstrap.cc:492] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: No bootstrap required, opened a new log
I20240502 07:59:03.698491 22538 tablet_bootstrap.cc:492] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: No bootstrap required, opened a new log
I20240502 07:59:03.698524 22537 ts_tablet_manager.cc:1367] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:59:03.698609 22538 ts_tablet_manager.cc:1367] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:03.701404 22538 raft_consensus.cc:348] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.701601 22538 raft_consensus.cc:374] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:03.701658 22538 raft_consensus.cc:729] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8afad6cf7ec44d88ff95b09ec01d077, State: Initialized, Role: FOLLOWER
I20240502 07:59:03.702073 22538 consensus_queue.cc:260] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.702008 22536 raft_consensus.cc:348] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.702232 22536 raft_consensus.cc:374] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:03.702132 22537 raft_consensus.cc:348] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.702296 22537 raft_consensus.cc:374] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:03.702291 22536 raft_consensus.cc:729] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bfecc3551b421d87d2abc1488bc4f6, State: Initialized, Role: FOLLOWER
I20240502 07:59:03.702353 22537 raft_consensus.cc:729] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223e69f38db94cfc945285235919ba74, State: Initialized, Role: FOLLOWER
I20240502 07:59:03.702742 22536 consensus_queue.cc:260] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.702744 22537 consensus_queue.cc:260] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.702891 22538 ts_tablet_manager.cc:1398] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 07:59:03.703256 22388 heartbeater.cc:503] Master 127.16.36.254:33429 was elected leader, sending a full tablet report...
I20240502 07:59:03.703550 22537 ts_tablet_manager.cc:1398] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:59:03.703550 22536 ts_tablet_manager.cc:1398] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20240502 07:59:03.703881 22257 heartbeater.cc:503] Master 127.16.36.254:33429 was elected leader, sending a full tablet report...
I20240502 07:59:03.703881 21995 heartbeater.cc:503] Master 127.16.36.254:33429 was elected leader, sending a full tablet report...
W20240502 07:59:03.704267 22389 tablet.cc:2369] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:59:03.755257 22258 tablet.cc:2369] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:03.849251 22544 raft_consensus.cc:482] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:03.849478 22544 raft_consensus.cc:504] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.850586 22544 leader_election.cc:288] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 223e69f38db94cfc945285235919ba74 (127.16.36.195:46531), e8afad6cf7ec44d88ff95b09ec01d077 (127.16.36.196:33515)
I20240502 07:59:03.854852 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e0220e4d3b4089b30c1b010652cd69" candidate_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223e69f38db94cfc945285235919ba74" is_pre_election: true
I20240502 07:59:03.854863 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e0220e4d3b4089b30c1b010652cd69" candidate_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" is_pre_election: true
I20240502 07:59:03.855144 22211 raft_consensus.cc:2454] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5bfecc3551b421d87d2abc1488bc4f6 in term 0.
I20240502 07:59:03.855180 22342 raft_consensus.cc:2454] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5bfecc3551b421d87d2abc1488bc4f6 in term 0.
I20240502 07:59:03.855646 21882 leader_election.cc:302] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5bfecc3551b421d87d2abc1488bc4f6, e8afad6cf7ec44d88ff95b09ec01d077; no voters:
I20240502 07:59:03.855917 22544 raft_consensus.cc:2781] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:03.856002 22544 raft_consensus.cc:482] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:03.856057 22544 raft_consensus.cc:3037] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:03.856963 22544 raft_consensus.cc:504] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.857331 22544 leader_election.cc:288] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [CANDIDATE]: Term 1 election: Requested vote from peers 223e69f38db94cfc945285235919ba74 (127.16.36.195:46531), e8afad6cf7ec44d88ff95b09ec01d077 (127.16.36.196:33515)
I20240502 07:59:03.857681 22342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e0220e4d3b4089b30c1b010652cd69" candidate_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8afad6cf7ec44d88ff95b09ec01d077"
I20240502 07:59:03.857676 22211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e0220e4d3b4089b30c1b010652cd69" candidate_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "223e69f38db94cfc945285235919ba74"
I20240502 07:59:03.857789 22342 raft_consensus.cc:3037] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:03.857801 22211 raft_consensus.cc:3037] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:03.858640 22211 raft_consensus.cc:2454] T 99e0220e4d3b4089b30c1b010652cd69 P 223e69f38db94cfc945285235919ba74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5bfecc3551b421d87d2abc1488bc4f6 in term 1.
I20240502 07:59:03.858647 22342 raft_consensus.cc:2454] T 99e0220e4d3b4089b30c1b010652cd69 P e8afad6cf7ec44d88ff95b09ec01d077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5bfecc3551b421d87d2abc1488bc4f6 in term 1.
I20240502 07:59:03.858901 21882 leader_election.cc:302] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5bfecc3551b421d87d2abc1488bc4f6, e8afad6cf7ec44d88ff95b09ec01d077; no voters:
I20240502 07:59:03.859059 22544 raft_consensus.cc:2781] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:03.859266 22544 raft_consensus.cc:686] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [term 1 LEADER]: Becoming Leader. State: Replica: a5bfecc3551b421d87d2abc1488bc4f6, State: Running, Role: LEADER
I20240502 07:59:03.859575 22544 consensus_queue.cc:237] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } }
I20240502 07:59:03.862473 21806 catalog_manager.cc:5484] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5bfecc3551b421d87d2abc1488bc4f6 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 } } peers { permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 } } peers { permanent_uuid: "a5bfecc3551b421d87d2abc1488bc4f6" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 40903 } } }
I20240502 07:59:03.873823 16531 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 07:59:03.875447 16531 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a5bfecc3551b421d87d2abc1488bc4f6 to finish bootstrapping
W20240502 07:59:03.881431 21996 tablet.cc:2369] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:03.881915 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 138e27ede9534845ad00e872a3c3df1c to finish bootstrapping
I20240502 07:59:03.888010 16531 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 223e69f38db94cfc945285235919ba74 to finish bootstrapping
I20240502 07:59:03.892815 16531 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e8afad6cf7ec44d88ff95b09ec01d077 to finish bootstrapping
I20240502 07:59:03.897536 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8ae13fc4213741f9a0d9ffde9f01b0f8 to finish bootstrapping
I20240502 07:59:04.080212 22126 heartbeater.cc:503] Master 127.16.36.254:33429 was elected leader, sending a full tablet report...
I20240502 07:59:04.087657 22453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.088773 22060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.089052 21929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:04.089349 22191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.101823 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.186561 22060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.189815 22191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.190016 22453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.191220 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.192297 21929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:04.292176 22544 consensus_queue.cc:1035] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "223e69f38db94cfc945285235919ba74" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 46531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:04.297923 22547 consensus_queue.cc:1035] T 99e0220e4d3b4089b30c1b010652cd69 P a5bfecc3551b421d87d2abc1488bc4f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8afad6cf7ec44d88ff95b09ec01d077" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:04.416538 22060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.417141 22453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.417732 22191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.418226 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.419118 21929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:04.509320 22060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.510679 22453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.511466 22191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.511893 21929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:04.511893 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.613358 22191 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.615078 22060 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.616600 21929 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:04.619968 22321 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.620918 22453 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:04.653993 22519 heartbeater.cc:503] Master 127.16.36.254:33429 was elected leader, sending a full tablet report...
I20240502 07:59:04.657153 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 21867
I20240502 07:59:04.664111 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 21998
I20240502 07:59:04.669557 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 22129
I20240502 07:59:04.674896 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 22260
I20240502 07:59:04.681051 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 22391
I20240502 07:59:04.686491 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 21776
2024-05-02T07:59:04Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/0 (2246 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2246 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/2
I20240502 07:59:04.709461 16531 test_util.cc:268] Using random seed: -99199665
I20240502 07:59:04.710245 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:59:04.710289 16531 ts_itest-base.cc:113] --------------
I20240502 07:59:04.710310 16531 ts_itest-base.cc:114] 7 tablet servers
I20240502 07:59:04.710327 16531 ts_itest-base.cc:115] 2 replicas per TS
I20240502 07:59:04.710353 16531 ts_itest-base.cc:116] --------------
2024-05-02T07:59:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:59:04Z Disabled control of system clock
I20240502 07:59:04.718122 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:45791
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:45791
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 07:59:04.824240 22730 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 07:59:04.824534 22730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:04.824590 22730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:04.824664 22730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:04.827989 22730 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:04.828086 22730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:59:04.828133 22730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:04.828171 22730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:59:04.828212 22730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:59:04.831959 22730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:45791
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:45791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.22730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:04.832949 22730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:04.834085 22730 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:04.842120 22736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.842828 22735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:04.843525 22738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:04.845002 22730 server_base.cc:1034] running on GCE node
I20240502 07:59:04.845697 22730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:04.846678 22730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:04.847842 22730 hybrid_clock.cc:648] HybridClock initialized: now 1714636744847813 us; error 45 us; skew 500 ppm
I20240502 07:59:04.849885 22730 webserver.cc:460] Webserver started at http://127.16.36.254:43543/ using document root <none> and password file <none>
I20240502 07:59:04.850454 22730 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:04.850517 22730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:04.850710 22730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:04.852351 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "5229ea5962df43b293bb46e794006e25"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-9sjq"
I20240502 07:59:04.852859 22730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "5229ea5962df43b293bb46e794006e25"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-9sjq"
I20240502 07:59:04.856196 22730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:04.858407 22744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:04.859429 22730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:04.859544 22730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "5229ea5962df43b293bb46e794006e25"
format_stamp: "Formatted at 2024-05-02 07:59:04 on dist-test-slave-9sjq"
I20240502 07:59:04.859637 22730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:04.873915 22730 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:04.874564 22730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:04.874723 22730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:04.881103 22730 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:45791
I20240502 07:59:04.881111 22796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:45791 every 8 connection(s)
I20240502 07:59:04.882045 22730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:59:04.883155 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 22730
I20240502 07:59:04.883291 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:59:04.884892 22797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:04.889887 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25: Bootstrap starting.
I20240502 07:59:04.892097 22797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:04.892891 22797 log.cc:824] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:04.894493 22797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25: No bootstrap required, opened a new log
I20240502 07:59:04.896968 22797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5229ea5962df43b293bb46e794006e25" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45791 } }
I20240502 07:59:04.897194 22797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:04.897258 22797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5229ea5962df43b293bb46e794006e25, State: Initialized, Role: FOLLOWER
I20240502 07:59:04.897662 22797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5229ea5962df43b293bb46e794006e25" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45791 } }
I20240502 07:59:04.897775 22797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:04.897848 22797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:04.897958 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:04.898685 22797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5229ea5962df43b293bb46e794006e25" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45791 } }
I20240502 07:59:04.898995 22797 leader_election.cc:302] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5229ea5962df43b293bb46e794006e25; no voters:
I20240502 07:59:04.899253 22797 leader_election.cc:288] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:04.899598 22802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:04.899799 22802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [term 1 LEADER]: Becoming Leader. State: Replica: 5229ea5962df43b293bb46e794006e25, State: Running, Role: LEADER
I20240502 07:59:04.900103 22802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5229ea5962df43b293bb46e794006e25" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45791 } }
I20240502 07:59:04.900336 22797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:04.901587 22803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5229ea5962df43b293bb46e794006e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5229ea5962df43b293bb46e794006e25" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45791 } } }
I20240502 07:59:04.901701 22803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:04.902036 22811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:04.901990 22804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5229ea5962df43b293bb46e794006e25. Latest consensus state: current_term: 1 leader_uuid: "5229ea5962df43b293bb46e794006e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5229ea5962df43b293bb46e794006e25" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 45791 } } }
I20240502 07:59:04.902143 22804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:04.904784 22811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:04.909896 22811 catalog_manager.cc:1347] Generated new cluster ID: aab6df9de97b490aa1d9eb9567f9c2cd
I20240502 07:59:04.909991 22811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:04.926584 22811 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:04.927538 22811 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:04.936045 22811 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 5229ea5962df43b293bb46e794006e25: Generated new TSK 0
I20240502 07:59:04.936908 22811 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:04.948150 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:05.058200 22821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:05.058431 22821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:05.058504 22821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:05.061687 22821 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:05.061782 22821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:05.061879 22821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 07:59:05.065788 22821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.22821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:05.066751 22821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:05.067945 22821 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:05.076606 22826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.077411 22827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.078132 22829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:05.079015 22821 server_base.cc:1034] running on GCE node
I20240502 07:59:05.079417 22821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:05.080039 22821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:05.081195 22821 hybrid_clock.cc:648] HybridClock initialized: now 1714636745081184 us; error 35 us; skew 500 ppm
I20240502 07:59:05.083060 22821 webserver.cc:460] Webserver started at http://127.16.36.193:37991/ using document root <none> and password file <none>
I20240502 07:59:05.083622 22821 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:05.083686 22821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:05.083901 22821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:05.085660 22821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5309ff82927d47ff824d67ad15631b0a"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.086160 22821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5309ff82927d47ff824d67ad15631b0a"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.089638 22821 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:05.091881 22835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.092967 22821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:05.093094 22821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5309ff82927d47ff824d67ad15631b0a"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.093190 22821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:05.114445 22821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:05.115149 22821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:05.115343 22821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:05.115900 22821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:05.116874 22821 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:05.116928 22821 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.116990 22821 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:05.117020 22821 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.125927 22821 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:46399
I20240502 07:59:05.126160 22948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:46399 every 8 connection(s)
I20240502 07:59:05.126964 22821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:05.134251 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 22821
I20240502 07:59:05.134353 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:59:05.136327 22949 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:05.136622 22949 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:05.136711 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:05.137180 22949 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:05.138617 22760 ts_manager.cc:194] Registered new tserver with Master: 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:05.139824 22760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:38463
W20240502 07:59:05.249526 22952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:05.249756 22952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:05.249800 22952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:05.253077 22952 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:05.253163 22952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:05.253234 22952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 07:59:05.257333 22952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.22952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:05.258296 22952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:05.259415 22952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:05.268162 22957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.268767 22960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.269629 22958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:05.270356 22952 server_base.cc:1034] running on GCE node
I20240502 07:59:05.270779 22952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:05.271355 22952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:05.272579 22952 hybrid_clock.cc:648] HybridClock initialized: now 1714636745272564 us; error 28 us; skew 500 ppm
I20240502 07:59:05.274493 22952 webserver.cc:460] Webserver started at http://127.16.36.194:40723/ using document root <none> and password file <none>
I20240502 07:59:05.275040 22952 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:05.275112 22952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:05.275317 22952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:05.277065 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "86a82caa98a74858bb3cb7b6b248f65e"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.277573 22952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "86a82caa98a74858bb3cb7b6b248f65e"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.280850 22952 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:05.282964 22966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.284010 22952 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:05.284123 22952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "86a82caa98a74858bb3cb7b6b248f65e"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.284207 22952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:05.296957 22952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:05.297643 22952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:05.297806 22952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:05.298354 22952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:05.299260 22952 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:05.299310 22952 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.299371 22952 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:05.299396 22952 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.307749 22952 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:38943
I20240502 07:59:05.307796 23079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:38943 every 8 connection(s)
I20240502 07:59:05.308728 22952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:59:05.312742 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 22952
I20240502 07:59:05.312875 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:59:05.315065 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:05.317893 23080 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:05.318117 23080 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:05.318605 23080 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:05.319628 22760 ts_manager.cc:194] Registered new tserver with Master: 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:05.320201 22760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:46601
W20240502 07:59:05.423523 23083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:05.423777 23083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:05.423856 23083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:05.426927 23083 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:05.427016 23083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:05.427116 23083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 07:59:05.430956 23083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.23083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:05.431903 23083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:05.433038 23083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:05.441251 23089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.442049 23088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.442873 23091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:05.443728 23083 server_base.cc:1034] running on GCE node
I20240502 07:59:05.444134 23083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:05.444767 23083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:05.445945 23083 hybrid_clock.cc:648] HybridClock initialized: now 1714636745445933 us; error 29 us; skew 500 ppm
I20240502 07:59:05.447749 23083 webserver.cc:460] Webserver started at http://127.16.36.195:33015/ using document root <none> and password file <none>
I20240502 07:59:05.448302 23083 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:05.448366 23083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:05.448594 23083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:05.450299 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d111283808d34d0d8671028dd2d826d5"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.450773 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d111283808d34d0d8671028dd2d826d5"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.454033 23083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:05.456167 23097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.457223 23083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:05.457345 23083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d111283808d34d0d8671028dd2d826d5"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.457432 23083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:05.472384 23083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:05.473066 23083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:05.473234 23083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:05.473755 23083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:05.474609 23083 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:05.474658 23083 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.474718 23083 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:05.474746 23083 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.483305 23083 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:33213
I20240502 07:59:05.483422 23210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:33213 every 8 connection(s)
I20240502 07:59:05.484513 23083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:05.491155 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 23083
I20240502 07:59:05.491264 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:59:05.493199 23211 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:05.493386 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:05.493407 23211 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:05.493856 23211 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:05.494825 22760 ts_manager.cc:194] Registered new tserver with Master: d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:05.495349 22760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:34951
W20240502 07:59:05.604331 23214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:05.604599 23214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:05.604660 23214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:05.607757 23214 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:05.607826 23214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:05.607924 23214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 07:59:05.611735 23214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.23214
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:05.612793 23214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:05.614141 23214 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:05.622502 23222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.623057 23219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.623987 23220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:05.624697 23214 server_base.cc:1034] running on GCE node
I20240502 07:59:05.625100 23214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:05.625690 23214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:05.626849 23214 hybrid_clock.cc:648] HybridClock initialized: now 1714636745626839 us; error 31 us; skew 500 ppm
I20240502 07:59:05.628816 23214 webserver.cc:460] Webserver started at http://127.16.36.196:37047/ using document root <none> and password file <none>
I20240502 07:59:05.629433 23214 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:05.629492 23214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:05.629679 23214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:05.631352 23214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2964136ef5e843c18c95afd711c81b09"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.631800 23214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2964136ef5e843c18c95afd711c81b09"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.635080 23214 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:05.637418 23228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.638409 23214 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:05.638520 23214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2964136ef5e843c18c95afd711c81b09"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.638613 23214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:05.669510 23214 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:05.670140 23214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:05.670279 23214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:05.670760 23214 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:05.671598 23214 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:05.671644 23214 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.671684 23214 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:05.671705 23214 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.680371 23214 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:42705
I20240502 07:59:05.680397 23341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:42705 every 8 connection(s)
I20240502 07:59:05.681318 23214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:05.689425 23342 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:05.689625 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 23214
I20240502 07:59:05.689677 23342 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:05.689754 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:59:05.690192 23342 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:05.691097 22761 ts_manager.cc:194] Registered new tserver with Master: 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705)
I20240502 07:59:05.691779 22761 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:48977
I20240502 07:59:05.691937 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:0
--local_ip_for_outbound_sockets=127.16.36.197
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:05.799417 23345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:05.799643 23345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:05.799687 23345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:05.802920 23345 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:05.802996 23345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:05.803095 23345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:59:05.806783 23345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.23345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:05.807736 23345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:05.808923 23345 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:05.817641 23350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.818393 23353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:05.819135 23351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:05.819965 23345 server_base.cc:1034] running on GCE node
I20240502 07:59:05.820359 23345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:05.820984 23345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:05.822144 23345 hybrid_clock.cc:648] HybridClock initialized: now 1714636745822130 us; error 30 us; skew 500 ppm
I20240502 07:59:05.829886 23345 webserver.cc:460] Webserver started at http://127.16.36.197:35625/ using document root <none> and password file <none>
I20240502 07:59:05.830495 23345 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:05.830564 23345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:05.830751 23345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:05.832556 23345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.833041 23345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.836490 23345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:05.838886 23359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.839944 23345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:05.840066 23345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:05.840162 23345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:05.856000 23345 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:05.856737 23345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:05.856906 23345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:05.857466 23345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:05.858381 23345 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:05.858436 23345 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.858500 23345 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:05.858528 23345 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:05.867377 23345 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:34707
I20240502 07:59:05.867513 23472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:34707 every 8 connection(s)
I20240502 07:59:05.868377 23345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:05.877405 23473 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:05.877651 23473 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:05.877760 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 23345
I20240502 07:59:05.877866 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:59:05.878221 23473 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:05.879149 22760 ts_manager.cc:194] Registered new tserver with Master: 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707)
I20240502 07:59:05.879694 22760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:36139
I20240502 07:59:05.879788 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.198:0
--local_ip_for_outbound_sockets=127.16.36.198
--webserver_interface=127.16.36.198
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:05.991559 23476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:05.991816 23476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:05.991891 23476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:05.995175 23476 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:05.995260 23476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:05.995359 23476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.198
I20240502 07:59:05.999224 23476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.198
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.23476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.198
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:06.000185 23476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:06.001399 23476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:06.010097 23481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:06.010687 23484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:06.011426 23482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:06.012271 23476 server_base.cc:1034] running on GCE node
I20240502 07:59:06.012668 23476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:06.013234 23476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:06.014397 23476 hybrid_clock.cc:648] HybridClock initialized: now 1714636746014384 us; error 27 us; skew 500 ppm
I20240502 07:59:06.016325 23476 webserver.cc:460] Webserver started at http://127.16.36.198:40437/ using document root <none> and password file <none>
I20240502 07:59:06.016904 23476 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:06.016968 23476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:06.017165 23476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:06.018810 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "12adf6ad821e4949acc3efc3f6a28404"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:06.019284 23476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "12adf6ad821e4949acc3efc3f6a28404"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:06.022619 23476 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:06.024757 23490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:06.025753 23476 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:59:06.025861 23476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "12adf6ad821e4949acc3efc3f6a28404"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:06.025955 23476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:06.037889 23476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:06.038533 23476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:06.038695 23476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:06.039222 23476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:06.040091 23476 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:06.040148 23476 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:06.040210 23476 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:06.040238 23476 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:06.048893 23476 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.198:38235
I20240502 07:59:06.048913 23603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.198:38235 every 8 connection(s)
I20240502 07:59:06.049845 23476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:06.055910 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 23476
I20240502 07:59:06.056042 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:59:06.058797 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.199:0
--local_ip_for_outbound_sockets=127.16.36.199
--webserver_interface=127.16.36.199
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:06.058995 23604 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:06.059260 23604 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:06.059757 23604 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:06.060921 22760 ts_manager.cc:194] Registered new tserver with Master: 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198:38235)
I20240502 07:59:06.061573 22760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.198:34103
I20240502 07:59:06.142187 22949 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
W20240502 07:59:06.171665 23607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:06.171957 23607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:06.172032 23607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:06.175431 23607 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:06.175529 23607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:06.175647 23607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.199
I20240502 07:59:06.179656 23607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.199
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.23607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.199
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:06.180729 23607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:06.181968 23607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:06.192417 23613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:06.192811 23615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:06.193997 23612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:06.194068 23607 server_base.cc:1034] running on GCE node
I20240502 07:59:06.194586 23607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:06.195281 23607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:06.196501 23607 hybrid_clock.cc:648] HybridClock initialized: now 1714636746196471 us; error 48 us; skew 500 ppm
I20240502 07:59:06.198755 23607 webserver.cc:460] Webserver started at http://127.16.36.199:40147/ using document root <none> and password file <none>
I20240502 07:59:06.199383 23607 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:06.199452 23607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:06.199651 23607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:06.201609 23607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e53ee470f3824340a097021b74b80353"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:06.202111 23607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e53ee470f3824340a097021b74b80353"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:06.205761 23607 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 07:59:06.208266 23621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:06.209561 23607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:06.209753 23607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e53ee470f3824340a097021b74b80353"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:06.209867 23607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:06.224013 23607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:06.224788 23607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:06.224964 23607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:06.225551 23607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:06.226475 23607 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:06.226524 23607 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:06.226572 23607 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:06.226603 23607 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:06.235728 23734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.199:34991 every 8 connection(s)
I20240502 07:59:06.235801 23607 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.199:34991
I20240502 07:59:06.236928 23607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:59:06.245508 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 23607
I20240502 07:59:06.245677 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 07:59:06.246970 23735 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:06.247272 23735 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:06.247833 23735 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:06.248891 22760 ts_manager.cc:194] Registered new tserver with Master: e53ee470f3824340a097021b74b80353 (127.16.36.199:34991)
I20240502 07:59:06.249440 22760 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.199:36077
I20240502 07:59:06.261188 16531 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 07:59:06.271497 22760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56244:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 07:59:06.290899 23538 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbca5be11d04fed987d979b02773974 (DEFAULT_TABLE table=TestTable [id=9d0f665e81544240bdb121395250b3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:06.290901 23276 tablet_service.cc:1467] Processing CreateTablet for tablet 2bbca5be11d04fed987d979b02773974 (DEFAULT_TABLE table=TestTable [id=9d0f665e81544240bdb121395250b3a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:06.291893 23538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbca5be11d04fed987d979b02773974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.291894 23276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bbca5be11d04fed987d979b02773974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:06.297422 23751 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Bootstrap starting.
I20240502 07:59:06.297436 23752 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap starting.
I20240502 07:59:06.299751 23752 tablet_bootstrap.cc:654] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:06.299747 23751 tablet_bootstrap.cc:654] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:06.300750 23751 log.cc:824] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:06.300786 23752 log.cc:824] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:06.302448 23752 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: No bootstrap required, opened a new log
I20240502 07:59:06.302448 23751 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: No bootstrap required, opened a new log
I20240502 07:59:06.302649 23751 ts_tablet_manager.cc:1367] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 07:59:06.302651 23752 ts_tablet_manager.cc:1367] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:59:06.305224 23751 raft_consensus.cc:348] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.305446 23751 raft_consensus.cc:374] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:06.305516 23751 raft_consensus.cc:729] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2964136ef5e843c18c95afd711c81b09, State: Initialized, Role: FOLLOWER
I20240502 07:59:06.305606 23752 raft_consensus.cc:348] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.305797 23752 raft_consensus.cc:374] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:06.305845 23752 raft_consensus.cc:729] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Initialized, Role: FOLLOWER
I20240502 07:59:06.306030 23751 consensus_queue.cc:260] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.306275 23752 consensus_queue.cc:260] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.307029 23342 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
I20240502 07:59:06.307124 23752 ts_tablet_manager.cc:1398] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 07:59:06.307181 23604 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
I20240502 07:59:06.307039 23751 ts_tablet_manager.cc:1398] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20240502 07:59:06.322113 23080 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
W20240502 07:59:06.432600 23343 tablet.cc:2369] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:06.478091 23756 raft_consensus.cc:482] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:06.478297 23756 raft_consensus.cc:504] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.479333 23756 leader_election.cc:288] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705)
I20240502 07:59:06.482743 23296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbca5be11d04fed987d979b02773974" candidate_uuid: "12adf6ad821e4949acc3efc3f6a28404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2964136ef5e843c18c95afd711c81b09" is_pre_election: true
I20240502 07:59:06.483062 23296 raft_consensus.cc:2454] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12adf6ad821e4949acc3efc3f6a28404 in term 0.
I20240502 07:59:06.483592 23493 leader_election.cc:302] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12adf6ad821e4949acc3efc3f6a28404, 2964136ef5e843c18c95afd711c81b09; no voters:
I20240502 07:59:06.483865 23756 raft_consensus.cc:2781] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:06.483958 23756 raft_consensus.cc:482] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:06.484005 23756 raft_consensus.cc:3037] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:06.485255 23756 raft_consensus.cc:504] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.485613 23756 leader_election.cc:288] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 1 election: Requested vote from peers 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705)
I20240502 07:59:06.486657 23296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbca5be11d04fed987d979b02773974" candidate_uuid: "12adf6ad821e4949acc3efc3f6a28404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2964136ef5e843c18c95afd711c81b09"
I20240502 07:59:06.486804 23296 raft_consensus.cc:3037] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:06.488093 23296 raft_consensus.cc:2454] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12adf6ad821e4949acc3efc3f6a28404 in term 1.
I20240502 07:59:06.488490 23493 leader_election.cc:302] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12adf6ad821e4949acc3efc3f6a28404, 2964136ef5e843c18c95afd711c81b09; no voters:
I20240502 07:59:06.488667 23756 raft_consensus.cc:2781] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:06.488890 23756 raft_consensus.cc:686] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 LEADER]: Becoming Leader. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Running, Role: LEADER
I20240502 07:59:06.489197 23756 consensus_queue.cc:237] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:06.491915 22760 catalog_manager.cc:5484] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198). New cstate: current_term: 1 leader_uuid: "12adf6ad821e4949acc3efc3f6a28404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:06.497277 23211 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
I20240502 07:59:06.521833 16531 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 07:59:06.523295 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5309ff82927d47ff824d67ad15631b0a to finish bootstrapping
I20240502 07:59:06.529747 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 86a82caa98a74858bb3cb7b6b248f65e to finish bootstrapping
I20240502 07:59:06.536042 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d111283808d34d0d8671028dd2d826d5 to finish bootstrapping
I20240502 07:59:06.542573 16531 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2964136ef5e843c18c95afd711c81b09 to finish bootstrapping
I20240502 07:59:06.547933 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 1f7c07da974747dc9d4d1a2c19c54f02 to finish bootstrapping
W20240502 07:59:06.550843 23605 tablet.cc:2369] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:06.554023 16531 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 12adf6ad821e4949acc3efc3f6a28404 to finish bootstrapping
I20240502 07:59:06.559190 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e53ee470f3824340a097021b74b80353 to finish bootstrapping
I20240502 07:59:06.565470 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 23214
I20240502 07:59:06.571391 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 23345
I20240502 07:59:06.576798 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 23476
I20240502 07:59:06.582451 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 23607
I20240502 07:59:10.339258 22760 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56244:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:10.353148 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 590e0019653d4f16b8d36cfa6f7ab8b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.353148 23145 tablet_service.cc:1467] Processing CreateTablet for tablet 590e0019653d4f16b8d36cfa6f7ab8b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.354092 23144 tablet_service.cc:1467] Processing CreateTablet for tablet fb030dd86b4a4e50a013a0fee3d8dcd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.354239 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590e0019653d4f16b8d36cfa6f7ab8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.354244 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590e0019653d4f16b8d36cfa6f7ab8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.353148 22882 tablet_service.cc:1467] Processing CreateTablet for tablet 30773ad41182415ea963b2f59efba756 (DEFAULT_TABLE table=rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.354620 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30773ad41182415ea963b2f59efba756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.354982 23014 tablet_service.cc:1467] Processing CreateTablet for tablet 30773ad41182415ea963b2f59efba756 (DEFAULT_TABLE table=rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.355810 23144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb030dd86b4a4e50a013a0fee3d8dcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.355861 23014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30773ad41182415ea963b2f59efba756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.360878 23013 tablet_service.cc:1467] Processing CreateTablet for tablet fb030dd86b4a4e50a013a0fee3d8dcd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:10.360954 23776 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:10.361238 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb030dd86b4a4e50a013a0fee3d8dcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:10.363590 23775 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:10.363806 23776 tablet_bootstrap.cc:654] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.364714 23777 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:10.364733 23776 log.cc:824] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:10.365864 23775 tablet_bootstrap.cc:654] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.366423 23777 tablet_bootstrap.cc:654] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.366587 23776 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:10.366662 23775 log.cc:824] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:10.366796 23776 ts_tablet_manager.cc:1367] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:59:10.367133 23777 log.cc:824] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:10.368526 23775 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:10.368726 23775 ts_tablet_manager.cc:1367] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:59:10.368854 23777 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:10.369012 23777 ts_tablet_manager.cc:1367] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 07:59:10.370014 23776 raft_consensus.cc:348] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.370191 23776 raft_consensus.cc:374] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.370222 23776 raft_consensus.cc:729] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:10.370636 23776 consensus_queue.cc:260] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.371587 23776 ts_tablet_manager.cc:1398] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240502 07:59:10.371976 23776 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:10.372654 23777 raft_consensus.cc:348] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.372654 23775 raft_consensus.cc:348] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.372855 23775 raft_consensus.cc:374] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.372855 23777 raft_consensus.cc:374] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.372906 23777 raft_consensus.cc:729] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:10.372917 23775 raft_consensus.cc:729] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:10.373020 23776 tablet_bootstrap.cc:654] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.373373 23777 consensus_queue.cc:260] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.373373 23775 consensus_queue.cc:260] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.374235 23776 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:10.374264 23777 ts_tablet_manager.cc:1398] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:59:10.374264 23775 ts_tablet_manager.cc:1398] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 07:59:10.374320 23776 ts_tablet_manager.cc:1367] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:10.374886 23776 raft_consensus.cc:348] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.375005 23776 raft_consensus.cc:374] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.375042 23776 raft_consensus.cc:729] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:10.375185 23776 consensus_queue.cc:260] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.375546 23777 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:10.375555 23776 ts_tablet_manager.cc:1398] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 07:59:10.376636 23777 tablet_bootstrap.cc:654] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.378026 23777 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:10.378119 23777 ts_tablet_manager.cc:1367] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:10.378765 23775 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:10.379022 23777 raft_consensus.cc:348] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.379148 23777 raft_consensus.cc:374] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.379196 23777 raft_consensus.cc:729] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:10.379345 23777 consensus_queue.cc:260] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.379616 23777 ts_tablet_manager.cc:1398] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:10.379928 23775 tablet_bootstrap.cc:654] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:10.381089 23775 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:10.381181 23775 ts_tablet_manager.cc:1367] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:10.381717 23775 raft_consensus.cc:348] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.381855 23775 raft_consensus.cc:374] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:10.381892 23775 raft_consensus.cc:729] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
W20240502 07:59:10.381953 22950 tablet.cc:2369] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:10.382021 23775 consensus_queue.cc:260] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.382328 23775 ts_tablet_manager.cc:1398] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 07:59:10.489326 23212 tablet.cc:2369] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:59:10.563668 23081 tablet.cc:2369] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:10.627807 23785 raft_consensus.cc:482] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:10.628010 23785 raft_consensus.cc:504] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.629046 23785 leader_election.cc:288] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:10.632719 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30773ad41182415ea963b2f59efba756" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:10.633014 23034 raft_consensus.cc:2454] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 0.
I20240502 07:59:10.633491 22836 leader_election.cc:302] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:10.633754 23785 raft_consensus.cc:2781] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:10.633833 23785 raft_consensus.cc:482] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:10.633872 23785 raft_consensus.cc:3037] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.634810 23785 raft_consensus.cc:504] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.635138 23785 leader_election.cc:288] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:10.635573 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30773ad41182415ea963b2f59efba756" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:10.635699 23034 raft_consensus.cc:3037] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.636713 23034 raft_consensus.cc:2454] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 1.
I20240502 07:59:10.637027 22836 leader_election.cc:302] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:10.637313 23785 raft_consensus.cc:2781] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:10.637434 23785 raft_consensus.cc:686] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Leader. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:10.637784 23785 consensus_queue.cc:237] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:10.640596 22759 catalog_manager.cc:5484] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5309ff82927d47ff824d67ad15631b0a (127.16.36.193). New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } }
I20240502 07:59:10.691434 23785 raft_consensus.cc:482] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:10.691577 23785 raft_consensus.cc:504] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.691991 23785 leader_election.cc:288] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:10.695508 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590e0019653d4f16b8d36cfa6f7ab8b6" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5" is_pre_election: true
I20240502 07:59:10.695818 23165 raft_consensus.cc:2454] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 0.
I20240502 07:59:10.696182 22839 leader_election.cc:302] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:10.696404 23785 raft_consensus.cc:2781] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:10.696493 23785 raft_consensus.cc:482] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:10.696545 23785 raft_consensus.cc:3037] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.697454 23785 raft_consensus.cc:504] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.697755 23785 leader_election.cc:288] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:10.698145 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "590e0019653d4f16b8d36cfa6f7ab8b6" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:10.698282 23165 raft_consensus.cc:3037] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.699152 23165 raft_consensus.cc:2454] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 1.
I20240502 07:59:10.699434 22839 leader_election.cc:302] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:10.699620 23785 raft_consensus.cc:2781] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:10.699713 23785 raft_consensus.cc:686] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Leader. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:10.699858 23785 consensus_queue.cc:237] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.701541 22759 catalog_manager.cc:5484] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5309ff82927d47ff824d67ad15631b0a (127.16.36.193). New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } }
I20240502 07:59:10.726931 23784 raft_consensus.cc:482] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:10.727085 23784 raft_consensus.cc:504] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.728042 23784 leader_election.cc:288] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:10.731933 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb030dd86b4a4e50a013a0fee3d8dcd2" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:10.732084 23034 raft_consensus.cc:2454] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 0.
I20240502 07:59:10.732434 23098 leader_election.cc:302] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:10.732743 23784 raft_consensus.cc:2781] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:10.732806 23784 raft_consensus.cc:482] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:10.732838 23784 raft_consensus.cc:3037] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.733686 23784 raft_consensus.cc:504] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.733975 23784 leader_election.cc:288] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:10.734342 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb030dd86b4a4e50a013a0fee3d8dcd2" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:10.734452 23034 raft_consensus.cc:3037] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:10.735193 23034 raft_consensus.cc:2454] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 1.
I20240502 07:59:10.735464 23098 leader_election.cc:302] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:10.735636 23784 raft_consensus.cc:2781] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:10.735852 23784 raft_consensus.cc:686] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:10.736148 23784 consensus_queue.cc:237] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:10.738883 22759 catalog_manager.cc:5484] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } }
W20240502 07:59:10.916561 23794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:10.916836 23794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:10.919970 23794 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:59:11.006683 23165 raft_consensus.cc:1261] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:11.007541 23781 consensus_queue.cc:1035] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:11.008776 23034 raft_consensus.cc:1261] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:11.009306 23033 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:11.009709 23785 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:11.010226 23792 consensus_queue.cc:1035] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 07:59:11.013626 23825 mvcc.cc:205] Tried to move back new op lower bound from 7023152132112969728 to 7023152130867482624. Current Snapshot: MvccSnapshot[applied={T|T < 7023152132112969728}]
I20240502 07:59:11.026147 23826 mvcc.cc:205] Tried to move back new op lower bound from 7023152132124000256 to 7023152131017523200. Current Snapshot: MvccSnapshot[applied={T|T < 7023152132124000256 or (T in {7023152132124000256})}]
I20240502 07:59:11.032608 23832 mvcc.cc:205] Tried to move back new op lower bound from 7023152132124000256 to 7023152131017523200. Current Snapshot: MvccSnapshot[applied={T|T < 7023152132124000256}]
INSERT report
rows total: 2000
time total: 81.0437 ms
time per row: 0.0405218 ms
I20240502 07:59:11.052583 22752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56244:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:11.058339 23013 tablet_service.cc:1467] Processing CreateTablet for tablet 029e2d974d514d6fb63ec3501eb86321 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.058426 22882 tablet_service.cc:1467] Processing CreateTablet for tablet 029e2d974d514d6fb63ec3501eb86321 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.058426 23145 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8ca72508214906a5dcaf459425afe4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.058704 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029e2d974d514d6fb63ec3501eb86321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.058720 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029e2d974d514d6fb63ec3501eb86321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.058720 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8ca72508214906a5dcaf459425afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.060750 23846 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:11.061594 23145 tablet_service.cc:1467] Processing CreateTablet for tablet 6e70ba6d33de4c7eaa320e2dda34ee70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.061650 23013 tablet_service.cc:1467] Processing CreateTablet for tablet 6e70ba6d33de4c7eaa320e2dda34ee70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.061868 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e70ba6d33de4c7eaa320e2dda34ee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.061887 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e70ba6d33de4c7eaa320e2dda34ee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.061895 22882 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8ca72508214906a5dcaf459425afe4 (DEFAULT_TABLE table=rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.062041 23846 tablet_bootstrap.cc:654] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.062134 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8ca72508214906a5dcaf459425afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.064195 23848 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:11.064568 23847 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:11.065129 23848 tablet_bootstrap.cc:654] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.065443 23847 tablet_bootstrap.cc:654] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.065512 23846 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:11.065599 23846 ts_tablet_manager.cc:1367] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:11.066013 23846 raft_consensus.cc:348] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.066109 23846 raft_consensus.cc:374] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.066144 23846 raft_consensus.cc:729] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.066298 23846 consensus_queue.cc:260] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.066525 23847 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:11.066525 23848 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:11.066583 23846 ts_tablet_manager.cc:1398] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.066630 23848 ts_tablet_manager.cc:1367] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:11.066630 23847 ts_tablet_manager.cc:1367] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:11.066764 23846 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:11.067066 23848 raft_consensus.cc:348] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.067057 23847 raft_consensus.cc:348] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.067152 23848 raft_consensus.cc:374] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.067152 23847 raft_consensus.cc:374] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.067184 23847 raft_consensus.cc:729] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.067184 23848 raft_consensus.cc:729] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.067313 23847 consensus_queue.cc:260] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.067337 23848 consensus_queue.cc:260] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.067577 23847 ts_tablet_manager.cc:1398] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:11.067656 23846 tablet_bootstrap.cc:654] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.067739 23847 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:11.068892 23847 tablet_bootstrap.cc:654] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.069365 23848 ts_tablet_manager.cc:1398] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:11.069540 23846 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:11.069633 23846 ts_tablet_manager.cc:1367] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:11.070080 23846 raft_consensus.cc:348] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.070199 23846 raft_consensus.cc:374] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.070240 23846 raft_consensus.cc:729] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.070379 23846 consensus_queue.cc:260] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.070633 23846 ts_tablet_manager.cc:1398] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.071123 23847 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:11.071202 23847 ts_tablet_manager.cc:1367] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:11.071220 23848 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:11.071630 23847 raft_consensus.cc:348] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.071753 23847 raft_consensus.cc:374] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.071786 23847 raft_consensus.cc:729] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.071908 23847 consensus_queue.cc:260] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.072130 23848 tablet_bootstrap.cc:654] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.072278 23847 ts_tablet_manager.cc:1398] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.073612 23848 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:11.073712 23848 ts_tablet_manager.cc:1367] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:11.074188 23848 raft_consensus.cc:348] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.074302 23848 raft_consensus.cc:374] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.074344 23848 raft_consensus.cc:729] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.074489 23848 consensus_queue.cc:260] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.074779 23848 ts_tablet_manager.cc:1398] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:11.131803 23792 raft_consensus.cc:482] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:11.131932 23792 raft_consensus.cc:504] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.132331 23792 leader_election.cc:288] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:11.132860 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e70ba6d33de4c7eaa320e2dda34ee70" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:11.133015 23034 raft_consensus.cc:2454] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 0.
I20240502 07:59:11.133379 23098 leader_election.cc:302] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.133599 23792 raft_consensus.cc:2781] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:11.133648 23792 raft_consensus.cc:482] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:11.133705 23792 raft_consensus.cc:3037] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.134599 23792 raft_consensus.cc:504] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.134874 23792 leader_election.cc:288] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:11.135246 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e70ba6d33de4c7eaa320e2dda34ee70" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:11.135360 23034 raft_consensus.cc:3037] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.136137 23034 raft_consensus.cc:2454] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 1.
I20240502 07:59:11.136476 23098 leader_election.cc:302] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.136699 23792 raft_consensus.cc:2781] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:11.136783 23792 raft_consensus.cc:686] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:11.136917 23792 consensus_queue.cc:237] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.138640 22757 catalog_manager.cc:5484] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } }
I20240502 07:59:11.181916 23830 raft_consensus.cc:482] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:11.182081 23830 raft_consensus.cc:504] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.183071 23830 leader_election.cc:288] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:11.186795 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029e2d974d514d6fb63ec3501eb86321" candidate_uuid: "86a82caa98a74858bb3cb7b6b248f65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a" is_pre_election: true
I20240502 07:59:11.187045 22903 raft_consensus.cc:2454] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a82caa98a74858bb3cb7b6b248f65e in term 0.
I20240502 07:59:11.187459 22969 leader_election.cc:302] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:11.187717 23830 raft_consensus.cc:2781] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:11.187779 23830 raft_consensus.cc:482] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:11.187831 23830 raft_consensus.cc:3037] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.189138 23830 raft_consensus.cc:504] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.189489 23830 leader_election.cc:288] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 election: Requested vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:11.189878 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "029e2d974d514d6fb63ec3501eb86321" candidate_uuid: "86a82caa98a74858bb3cb7b6b248f65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:11.189996 22903 raft_consensus.cc:3037] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.190863 22903 raft_consensus.cc:2454] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a82caa98a74858bb3cb7b6b248f65e in term 1.
I20240502 07:59:11.191167 22969 leader_election.cc:302] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:11.191375 23830 raft_consensus.cc:2781] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:11.191470 23830 raft_consensus.cc:686] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Becoming Leader. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Running, Role: LEADER
I20240502 07:59:11.191787 23830 consensus_queue.cc:237] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.193914 22757 catalog_manager.cc:5484] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } }
I20240502 07:59:11.349064 23792 raft_consensus.cc:482] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:11.349260 23792 raft_consensus.cc:504] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.349730 23792 leader_election.cc:288] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:11.353333 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8ca72508214906a5dcaf459425afe4" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a" is_pre_election: true
I20240502 07:59:11.353514 22903 raft_consensus.cc:2454] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 0.
I20240502 07:59:11.353850 23100 leader_election.cc:302] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.354056 23792 raft_consensus.cc:2781] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:11.354125 23792 raft_consensus.cc:482] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:11.354179 23792 raft_consensus.cc:3037] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.355017 23792 raft_consensus.cc:504] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.355342 23792 leader_election.cc:288] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Requested vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:11.355772 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8ca72508214906a5dcaf459425afe4" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:11.355896 22903 raft_consensus.cc:3037] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.356712 22903 raft_consensus.cc:2454] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 1.
I20240502 07:59:11.357008 23100 leader_election.cc:302] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.357187 23792 raft_consensus.cc:2781] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:11.357293 23792 raft_consensus.cc:686] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:11.357432 23792 consensus_queue.cc:237] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:11.359040 22757 catalog_manager.cc:5484] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } }
W20240502 07:59:11.544966 23860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:11.545327 23860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:11.548678 23860 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 07:59:11.636729 23034 raft_consensus.cc:1261] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:11.637324 22903 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:11.637691 23792 consensus_queue.cc:1035] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:11.638015 23784 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:11.639091 23829 consensus_queue.cc:1035] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 90.3289 ms
time per row: 0.0451644 ms
I20240502 07:59:11.684293 22752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56244:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:11.690294 23013 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf181a9a54e4a05a354ffbcaf020f4c (DEFAULT_TABLE table=rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.690285 22883 tablet_service.cc:1467] Processing CreateTablet for tablet eb6626f239b34d5da46b58f4e849145d (DEFAULT_TABLE table=rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.690440 22882 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf181a9a54e4a05a354ffbcaf020f4c (DEFAULT_TABLE table=rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.690609 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6626f239b34d5da46b58f4e849145d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.690608 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf181a9a54e4a05a354ffbcaf020f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.690825 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf181a9a54e4a05a354ffbcaf020f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.691417 23145 tablet_service.cc:1467] Processing CreateTablet for tablet eb6626f239b34d5da46b58f4e849145d (DEFAULT_TABLE table=rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.691700 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6626f239b34d5da46b58f4e849145d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.693086 23144 tablet_service.cc:1467] Processing CreateTablet for tablet e91a0e50f7ce40e496812b9d415b2fa4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.693331 23144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91a0e50f7ce40e496812b9d415b2fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.693945 23914 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:11.694054 23913 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:11.694875 23914 tablet_bootstrap.cc:654] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.694875 23913 tablet_bootstrap.cc:654] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.695433 23013 tablet_service.cc:1467] Processing CreateTablet for tablet e91a0e50f7ce40e496812b9d415b2fa4 (DEFAULT_TABLE table=rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:11.695647 23912 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:11.695712 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91a0e50f7ce40e496812b9d415b2fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:11.696578 23912 tablet_bootstrap.cc:654] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.697367 23913 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:11.697458 23913 ts_tablet_manager.cc:1367] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:11.697633 23912 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:11.697712 23912 ts_tablet_manager.cc:1367] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:11.698138 23913 raft_consensus.cc:348] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.698149 23912 raft_consensus.cc:348] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.698267 23913 raft_consensus.cc:374] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.698272 23912 raft_consensus.cc:374] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.698309 23913 raft_consensus.cc:729] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.698309 23912 raft_consensus.cc:729] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.698431 23913 consensus_queue.cc:260] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.698439 23912 consensus_queue.cc:260] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.698503 23914 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:11.698582 23914 ts_tablet_manager.cc:1367] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:11.698724 23912 ts_tablet_manager.cc:1398] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.698750 23913 ts_tablet_manager.cc:1398] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.698864 23912 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:11.698930 23913 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:11.699725 23912 tablet_bootstrap.cc:654] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.699839 23913 tablet_bootstrap.cc:654] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.700830 23914 raft_consensus.cc:348] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.700961 23914 raft_consensus.cc:374] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.701000 23914 raft_consensus.cc:729] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.701329 23914 consensus_queue.cc:260] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.701465 23913 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:11.701542 23913 ts_tablet_manager.cc:1367] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:11.701644 23914 ts_tablet_manager.cc:1398] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:11.701802 23914 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:11.701964 23913 raft_consensus.cc:348] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.702050 23913 raft_consensus.cc:374] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.702078 23913 raft_consensus.cc:729] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.702236 23913 consensus_queue.cc:260] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.702564 23913 ts_tablet_manager.cc:1398] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.703474 23914 tablet_bootstrap.cc:654] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:11.703485 23912 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:11.703583 23912 ts_tablet_manager.cc:1367] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:11.704092 23912 raft_consensus.cc:348] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.704195 23912 raft_consensus.cc:374] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.704229 23912 raft_consensus.cc:729] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.704372 23912 consensus_queue.cc:260] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.704635 23914 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:11.704643 23912 ts_tablet_manager.cc:1398] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.704731 23914 ts_tablet_manager.cc:1367] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:11.705116 23914 raft_consensus.cc:348] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.705215 23914 raft_consensus.cc:374] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:11.705255 23914 raft_consensus.cc:729] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:11.705382 23914 consensus_queue.cc:260] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.705749 23914 ts_tablet_manager.cc:1398] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:11.793004 23785 raft_consensus.cc:482] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:11.793233 23785 raft_consensus.cc:504] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.793608 23785 leader_election.cc:288] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:11.794137 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6626f239b34d5da46b58f4e849145d" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5" is_pre_election: true
I20240502 07:59:11.794277 23165 raft_consensus.cc:2454] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 0.
I20240502 07:59:11.794641 22839 leader_election.cc:302] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.794847 23785 raft_consensus.cc:2781] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:11.794934 23785 raft_consensus.cc:482] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:11.794991 23785 raft_consensus.cc:3037] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.796026 23785 raft_consensus.cc:504] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.796339 23785 leader_election.cc:288] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:11.796700 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb6626f239b34d5da46b58f4e849145d" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:11.796813 23165 raft_consensus.cc:3037] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.797669 23165 raft_consensus.cc:2454] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 1.
I20240502 07:59:11.797963 22839 leader_election.cc:302] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.798218 23785 raft_consensus.cc:2781] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:11.798409 23785 raft_consensus.cc:686] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Leader. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:11.798556 23785 consensus_queue.cc:237] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.800347 22752 catalog_manager.cc:5484] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5309ff82927d47ff824d67ad15631b0a (127.16.36.193). New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } }
I20240502 07:59:11.841027 23785 raft_consensus.cc:482] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:11.841184 23785 raft_consensus.cc:504] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.841580 23785 leader_election.cc:288] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:11.842126 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf181a9a54e4a05a354ffbcaf020f4c" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:11.842270 23034 raft_consensus.cc:2454] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 0.
I20240502 07:59:11.842629 22836 leader_election.cc:302] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:11.842861 23785 raft_consensus.cc:2781] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:11.842932 23785 raft_consensus.cc:482] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:11.842973 23785 raft_consensus.cc:3037] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.843771 23785 raft_consensus.cc:504] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.844055 23785 leader_election.cc:288] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:11.844533 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf181a9a54e4a05a354ffbcaf020f4c" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:11.844641 23034 raft_consensus.cc:3037] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.845443 23034 raft_consensus.cc:2454] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 1.
I20240502 07:59:11.845746 22836 leader_election.cc:302] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:11.845938 23785 raft_consensus.cc:2781] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:11.846050 23785 raft_consensus.cc:686] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Leader. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:11.846262 23785 consensus_queue.cc:237] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:11.847962 22752 catalog_manager.cc:5484] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5309ff82927d47ff824d67ad15631b0a (127.16.36.193). New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } }
I20240502 07:59:11.909231 23792 raft_consensus.cc:482] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:11.909376 23792 raft_consensus.cc:504] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.909766 23792 leader_election.cc:288] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:11.910316 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91a0e50f7ce40e496812b9d415b2fa4" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:11.910454 23034 raft_consensus.cc:2454] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 0.
I20240502 07:59:11.910837 23098 leader_election.cc:302] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.911026 23792 raft_consensus.cc:2781] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:11.911091 23792 raft_consensus.cc:482] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:11.911126 23792 raft_consensus.cc:3037] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.911934 23792 raft_consensus.cc:504] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.912204 23792 leader_election.cc:288] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:11.912705 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91a0e50f7ce40e496812b9d415b2fa4" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:11.912838 23034 raft_consensus.cc:3037] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:11.913717 23034 raft_consensus.cc:2454] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 1.
I20240502 07:59:11.914093 23098 leader_election.cc:302] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:11.914278 23792 raft_consensus.cc:2781] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:11.914376 23792 raft_consensus.cc:686] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:11.914536 23792 consensus_queue.cc:237] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:11.916186 22752 catalog_manager.cc:5484] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } }
W20240502 07:59:12.041289 23925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:12.041555 23925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:12.044912 23925 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:59:12.135392 23165 raft_consensus.cc:1261] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:12.135960 23033 raft_consensus.cc:1261] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:12.136119 23034 raft_consensus.cc:1261] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:12.136411 23792 consensus_queue.cc:1035] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:12.136684 23921 consensus_queue.cc:1035] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:12.136698 23785 consensus_queue.cc:1035] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.4343 ms
time per row: 0.0497172 ms
I20240502 07:59:12.188766 22753 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56244:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:12.194959 22882 tablet_service.cc:1467] Processing CreateTablet for tablet a207b20592e74478beb63c8000b3b1b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.195044 23013 tablet_service.cc:1467] Processing CreateTablet for tablet a207b20592e74478beb63c8000b3b1b0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.195330 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a207b20592e74478beb63c8000b3b1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.195330 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a207b20592e74478beb63c8000b3b1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.195544 23144 tablet_service.cc:1467] Processing CreateTablet for tablet 15fe5361ff8d4e36b6e59b48813f4aaf (DEFAULT_TABLE table=rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.195887 23144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fe5361ff8d4e36b6e59b48813f4aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.195976 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 15fe5361ff8d4e36b6e59b48813f4aaf (DEFAULT_TABLE table=rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.196190 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fe5361ff8d4e36b6e59b48813f4aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.197624 23913 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:12.197794 23912 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:12.195544 23145 tablet_service.cc:1467] Processing CreateTablet for tablet f4ecc981a63c4488aaa9c58a272b6882 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.198298 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ecc981a63c4488aaa9c58a272b6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.198532 23912 tablet_bootstrap.cc:654] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.198532 23913 tablet_bootstrap.cc:654] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.199143 23914 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:12.199823 23913 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:12.199900 23913 ts_tablet_manager.cc:1367] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:12.199997 23914 tablet_bootstrap.cc:654] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.200383 23913 raft_consensus.cc:348] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.200537 23913 raft_consensus.cc:374] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.200768 23913 raft_consensus.cc:729] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.200961 23913 consensus_queue.cc:260] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.201243 23913 ts_tablet_manager.cc:1398] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:12.201335 23013 tablet_service.cc:1467] Processing CreateTablet for tablet f4ecc981a63c4488aaa9c58a272b6882 (DEFAULT_TABLE table=rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.201418 23913 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:12.201581 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ecc981a63c4488aaa9c58a272b6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.202257 23913 tablet_bootstrap.cc:654] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.203356 23913 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:12.203457 23913 ts_tablet_manager.cc:1367] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:12.203925 23913 raft_consensus.cc:348] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.204027 23913 raft_consensus.cc:374] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.204063 23913 raft_consensus.cc:729] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.204176 23913 consensus_queue.cc:260] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.204205 23914 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:12.204260 23912 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:12.204285 23914 ts_tablet_manager.cc:1367] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:12.204334 23912 ts_tablet_manager.cc:1367] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240502 07:59:12.204566 23913 ts_tablet_manager.cc:1398] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.204775 23912 raft_consensus.cc:348] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.204775 23914 raft_consensus.cc:348] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.204869 23912 raft_consensus.cc:374] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.204876 23914 raft_consensus.cc:374] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.204905 23912 raft_consensus.cc:729] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.204913 23914 raft_consensus.cc:729] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.205030 23912 consensus_queue.cc:260] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.205044 23914 consensus_queue.cc:260] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.205294 23912 ts_tablet_manager.cc:1398] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.205320 23914 ts_tablet_manager.cc:1398] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.205459 23912 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:12.205475 23914 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:12.206321 23914 tablet_bootstrap.cc:654] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.206321 23912 tablet_bootstrap.cc:654] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.207793 23912 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:12.207932 23912 ts_tablet_manager.cc:1367] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:12.208371 23912 raft_consensus.cc:348] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.208557 23912 raft_consensus.cc:374] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.208616 23912 raft_consensus.cc:729] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.208761 23912 consensus_queue.cc:260] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.208982 23914 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:12.209064 23914 ts_tablet_manager.cc:1367] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:12.209066 23912 ts_tablet_manager.cc:1398] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.209492 23914 raft_consensus.cc:348] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.209625 23914 raft_consensus.cc:374] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.209666 23914 raft_consensus.cc:729] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.209793 23914 consensus_queue.cc:260] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.210080 23914 ts_tablet_manager.cc:1398] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.241133 23901 raft_consensus.cc:482] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:12.241266 23901 raft_consensus.cc:504] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.241677 23901 leader_election.cc:288] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:12.245255 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ecc981a63c4488aaa9c58a272b6882" candidate_uuid: "86a82caa98a74858bb3cb7b6b248f65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5" is_pre_election: true
I20240502 07:59:12.245401 23165 raft_consensus.cc:2454] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a82caa98a74858bb3cb7b6b248f65e in term 0.
I20240502 07:59:12.245707 22970 leader_election.cc:302] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:12.245923 23901 raft_consensus.cc:2781] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:12.245990 23901 raft_consensus.cc:482] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:12.246019 23901 raft_consensus.cc:3037] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.246876 23901 raft_consensus.cc:504] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.247184 23901 leader_election.cc:288] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:12.247632 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4ecc981a63c4488aaa9c58a272b6882" candidate_uuid: "86a82caa98a74858bb3cb7b6b248f65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:12.247779 23165 raft_consensus.cc:3037] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.248611 23165 raft_consensus.cc:2454] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a82caa98a74858bb3cb7b6b248f65e in term 1.
I20240502 07:59:12.248889 22970 leader_election.cc:302] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:12.249051 23901 raft_consensus.cc:2781] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:12.249217 23901 raft_consensus.cc:686] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Becoming Leader. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Running, Role: LEADER
I20240502 07:59:12.249377 23901 consensus_queue.cc:237] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } }
I20240502 07:59:12.250509 23921 raft_consensus.cc:482] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:12.250628 23921 raft_consensus.cc:504] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.250972 23921 leader_election.cc:288] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:12.251201 22753 catalog_manager.cc:5484] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } }
I20240502 07:59:12.251312 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5" is_pre_election: true
I20240502 07:59:12.251416 23165 raft_consensus.cc:2454] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 0.
I20240502 07:59:12.251734 22839 leader_election.cc:302] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:12.251919 23921 raft_consensus.cc:2781] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:12.251981 23921 raft_consensus.cc:482] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:12.252038 23921 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.253389 23921 raft_consensus.cc:504] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.253682 23921 leader_election.cc:288] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:12.254153 23165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:12.254277 23165 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.255012 23165 raft_consensus.cc:2454] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 1.
I20240502 07:59:12.255306 22839 leader_election.cc:302] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:12.255457 23921 raft_consensus.cc:2781] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:12.255545 23921 raft_consensus.cc:686] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Leader. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:12.255692 23921 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.257251 22753 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5309ff82927d47ff824d67ad15631b0a (127.16.36.193). New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } }
I20240502 07:59:12.388427 23921 raft_consensus.cc:482] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:12.388621 23921 raft_consensus.cc:504] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.389091 23921 leader_election.cc:288] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:12.389520 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a207b20592e74478beb63c8000b3b1b0" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:12.389655 23034 raft_consensus.cc:2454] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 0.
I20240502 07:59:12.389954 22836 leader_election.cc:302] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:12.390106 23921 raft_consensus.cc:2781] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:12.390178 23921 raft_consensus.cc:482] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:12.390215 23921 raft_consensus.cc:3037] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.391249 23921 raft_consensus.cc:504] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.391499 23921 leader_election.cc:288] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:12.391850 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a207b20592e74478beb63c8000b3b1b0" candidate_uuid: "5309ff82927d47ff824d67ad15631b0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:12.391965 23034 raft_consensus.cc:3037] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.392777 23034 raft_consensus.cc:2454] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5309ff82927d47ff824d67ad15631b0a in term 1.
I20240502 07:59:12.393170 22836 leader_election.cc:302] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:12.393353 23921 raft_consensus.cc:2781] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:12.393436 23921 raft_consensus.cc:686] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Leader. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:12.393579 23921 consensus_queue.cc:237] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.395270 22753 catalog_manager.cc:5484] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: term changed from 0 to 1, leader changed from <none> to 5309ff82927d47ff824d67ad15631b0a (127.16.36.193). New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } }
W20240502 07:59:12.548043 23967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:12.548312 23967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:12.551532 23967 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 07:59:12.629920 23034 raft_consensus.cc:1261] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:12.629920 23165 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:12.630394 23785 consensus_queue.cc:1035] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:12.630712 23785 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:12.631707 23165 raft_consensus.cc:1261] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:12.632196 23963 consensus_queue.cc:1035] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 82.7882 ms
time per row: 0.0413941 ms
I20240502 07:59:12.678514 22752 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56244:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:12.684319 23013 tablet_service.cc:1467] Processing CreateTablet for tablet 37b107413a8b44c2a72967b33ce4f0ee (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.684396 22883 tablet_service.cc:1467] Processing CreateTablet for tablet 37b107413a8b44c2a72967b33ce4f0ee (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.684614 22882 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6799e8df114d9c97cf5aafe3233ce2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.684724 22883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b107413a8b44c2a72967b33ce4f0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.684727 23013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b107413a8b44c2a72967b33ce4f0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.684948 22882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6799e8df114d9c97cf5aafe3233ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.685168 23014 tablet_service.cc:1467] Processing CreateTablet for tablet aa8e72ea69b147f4925c7cb6cb8ad24d (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.685384 23014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8e72ea69b147f4925c7cb6cb8ad24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.685319 23145 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6799e8df114d9c97cf5aafe3233ce2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.685565 23145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6799e8df114d9c97cf5aafe3233ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.687026 23912 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:12.687227 23913 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:12.687551 23144 tablet_service.cc:1467] Processing CreateTablet for tablet aa8e72ea69b147f4925c7cb6cb8ad24d (DEFAULT_TABLE table=rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:12.687727 23914 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:12.687795 23144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8e72ea69b147f4925c7cb6cb8ad24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:12.687909 23912 tablet_bootstrap.cc:654] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.688024 23913 tablet_bootstrap.cc:654] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.688696 23914 tablet_bootstrap.cc:654] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.689462 23912 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:12.689549 23912 ts_tablet_manager.cc:1367] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:12.689985 23912 raft_consensus.cc:348] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.690100 23912 raft_consensus.cc:374] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.690141 23912 raft_consensus.cc:729] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.690356 23912 consensus_queue.cc:260] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.690462 23914 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:12.690532 23914 ts_tablet_manager.cc:1367] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:12.690668 23912 ts_tablet_manager.cc:1398] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.690840 23912 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: Bootstrap starting.
I20240502 07:59:12.690977 23914 raft_consensus.cc:348] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.691087 23914 raft_consensus.cc:374] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.691129 23914 raft_consensus.cc:729] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.691279 23914 consensus_queue.cc:260] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.691527 23914 ts_tablet_manager.cc:1398] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.691697 23914 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Bootstrap starting.
I20240502 07:59:12.691705 23912 tablet_bootstrap.cc:654] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.693100 23912 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: No bootstrap required, opened a new log
I20240502 07:59:12.693137 23913 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:12.693187 23912 ts_tablet_manager.cc:1367] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:12.693207 23913 ts_tablet_manager.cc:1367] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 07:59:12.693603 23912 raft_consensus.cc:348] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.693616 23913 raft_consensus.cc:348] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.693698 23912 raft_consensus.cc:374] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.693698 23913 raft_consensus.cc:374] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.693730 23913 raft_consensus.cc:729] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.693730 23912 raft_consensus.cc:729] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.693850 23913 consensus_queue.cc:260] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.693876 23912 consensus_queue.cc:260] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.694104 23913 ts_tablet_manager.cc:1398] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:12.694221 23912 ts_tablet_manager.cc:1398] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:12.694279 23913 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: Bootstrap starting.
I20240502 07:59:12.695518 23914 tablet_bootstrap.cc:654] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.696831 23914 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: No bootstrap required, opened a new log
I20240502 07:59:12.696882 23913 tablet_bootstrap.cc:654] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:12.696915 23914 ts_tablet_manager.cc:1367] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:12.697317 23914 raft_consensus.cc:348] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.697433 23914 raft_consensus.cc:374] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.697466 23914 raft_consensus.cc:729] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.697592 23914 consensus_queue.cc:260] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:12.697899 23914 ts_tablet_manager.cc:1398] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:12.698236 23913 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: No bootstrap required, opened a new log
I20240502 07:59:12.698333 23913 ts_tablet_manager.cc:1367] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 07:59:12.698797 23913 raft_consensus.cc:348] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.698907 23913 raft_consensus.cc:374] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:12.698958 23913 raft_consensus.cc:729] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Initialized, Role: FOLLOWER
I20240502 07:59:12.699095 23913 consensus_queue.cc:260] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.699401 23913 ts_tablet_manager.cc:1398] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:12.764849 23792 raft_consensus.cc:482] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:12.764982 23792 raft_consensus.cc:504] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.765342 23792 leader_election.cc:288] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:12.765851 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6799e8df114d9c97cf5aafe3233ce2" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a" is_pre_election: true
I20240502 07:59:12.766018 22903 raft_consensus.cc:2454] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 0.
I20240502 07:59:12.766410 23100 leader_election.cc:302] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:12.766654 23792 raft_consensus.cc:2781] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:12.766708 23792 raft_consensus.cc:482] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:12.766736 23792 raft_consensus.cc:3037] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.767526 23792 raft_consensus.cc:504] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.767814 23792 leader_election.cc:288] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Requested vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:12.768229 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6799e8df114d9c97cf5aafe3233ce2" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:12.768343 22903 raft_consensus.cc:3037] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.769146 22903 raft_consensus.cc:2454] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 1.
I20240502 07:59:12.769466 23100 leader_election.cc:302] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:12.769703 23792 raft_consensus.cc:2781] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:12.769891 23792 raft_consensus.cc:686] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:12.770032 23792 consensus_queue.cc:237] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.771729 22760 catalog_manager.cc:5484] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } }
I20240502 07:59:12.882226 23963 raft_consensus.cc:482] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:12.882372 23963 raft_consensus.cc:504] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.882759 23963 leader_election.cc:288] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:12.883224 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b107413a8b44c2a72967b33ce4f0ee" candidate_uuid: "86a82caa98a74858bb3cb7b6b248f65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a" is_pre_election: true
I20240502 07:59:12.883383 22903 raft_consensus.cc:2454] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86a82caa98a74858bb3cb7b6b248f65e in term 0.
I20240502 07:59:12.883780 22969 leader_election.cc:302] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:12.883973 23963 raft_consensus.cc:2781] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:12.884049 23963 raft_consensus.cc:482] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:12.884084 23963 raft_consensus.cc:3037] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.885182 23963 raft_consensus.cc:504] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.885479 23963 leader_election.cc:288] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 election: Requested vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:12.885924 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37b107413a8b44c2a72967b33ce4f0ee" candidate_uuid: "86a82caa98a74858bb3cb7b6b248f65e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:12.886078 22903 raft_consensus.cc:3037] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:12.886932 22903 raft_consensus.cc:2454] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86a82caa98a74858bb3cb7b6b248f65e in term 1.
I20240502 07:59:12.887264 22969 leader_election.cc:302] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, 86a82caa98a74858bb3cb7b6b248f65e; no voters:
I20240502 07:59:12.887509 23963 raft_consensus.cc:2781] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:12.887609 23963 raft_consensus.cc:686] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Becoming Leader. State: Replica: 86a82caa98a74858bb3cb7b6b248f65e, State: Running, Role: LEADER
I20240502 07:59:12.887760 23963 consensus_queue.cc:237] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } }
I20240502 07:59:12.889379 22760 catalog_manager.cc:5484] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: term changed from 0 to 1, leader changed from <none> to 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } }
I20240502 07:59:13.025707 23792 raft_consensus.cc:482] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:13.025866 23792 raft_consensus.cc:504] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:13.026249 23792 leader_election.cc:288] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:13.026881 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8e72ea69b147f4925c7cb6cb8ad24d" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e" is_pre_election: true
I20240502 07:59:13.027041 23034 raft_consensus.cc:2454] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 0.
I20240502 07:59:13.027417 23098 leader_election.cc:302] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:13.027628 23792 raft_consensus.cc:2781] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:13.027693 23792 raft_consensus.cc:482] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:13.027724 23792 raft_consensus.cc:3037] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:13.028627 23792 raft_consensus.cc:504] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:13.028939 23792 leader_election.cc:288] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Requested vote from peers 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:13.029414 23034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8e72ea69b147f4925c7cb6cb8ad24d" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:13.029534 23034 raft_consensus.cc:3037] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:13.030367 23034 raft_consensus.cc:2454] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 1.
I20240502 07:59:13.030721 23098 leader_election.cc:302] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 86a82caa98a74858bb3cb7b6b248f65e, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:13.030982 23792 raft_consensus.cc:2781] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:13.031145 23792 raft_consensus.cc:686] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:13.031535 23792 consensus_queue.cc:237] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } }
I20240502 07:59:13.033154 22760 catalog_manager.cc:5484] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } }
W20240502 07:59:13.173094 24008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:13.173457 24008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:13.176972 24008 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 07:59:13.261022 23034 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:13.261072 22903 raft_consensus.cc:1261] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:13.261495 24004 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:13.262358 23963 consensus_queue.cc:1035] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:13.262418 22903 raft_consensus.cc:1261] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:13.262815 23792 consensus_queue.cc:1035] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 79.8411 ms
time per row: 0.0399206 ms
I20240502 07:59:13.304076 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:42705
--local_ip_for_outbound_sockets=127.16.36.196
--tserver_master_addrs=127.16.36.254:45791
--webserver_port=37047
--webserver_interface=127.16.36.196
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:13.417008 24059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:13.417263 24059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:13.417312 24059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:13.420508 24059 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:13.420576 24059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:13.420675 24059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 07:59:13.424439 24059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:42705
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=37047
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.24059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:13.425420 24059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:13.426576 24059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:59:13.435897 24059 server_base.cc:1034] running on GCE node
W20240502 07:59:13.436421 24067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:13.437139 24065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:13.437775 24064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:13.438311 24059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:13.438941 24059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:13.440155 24059 hybrid_clock.cc:648] HybridClock initialized: now 1714636753440122 us; error 46 us; skew 500 ppm
I20240502 07:59:13.442170 24059 webserver.cc:460] Webserver started at http://127.16.36.196:37047/ using document root <none> and password file <none>
I20240502 07:59:13.442730 24059 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:13.442801 24059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:13.446451 24059 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240502 07:59:13.448748 24073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:13.450143 24059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:13.450277 24059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2964136ef5e843c18c95afd711c81b09"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:13.450738 24059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:13.481067 24059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:13.481765 24059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:13.481937 24059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:13.482496 24059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:13.483637 24080 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 07:59:13.487605 24059 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 07:59:13.487691 24059 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240502 07:59:13.487776 24059 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 07:59:13.489421 24059 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 07:59:13.489468 24059 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:13.489746 24080 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Bootstrap starting.
I20240502 07:59:13.500344 24059 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:42705
I20240502 07:59:13.500597 24187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:42705 every 8 connection(s)
I20240502 07:59:13.501605 24059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:13.504912 24080 log.cc:824] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:13.506392 24080 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:13.506723 24080 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Bootstrap complete.
I20240502 07:59:13.508567 24080 ts_tablet_manager.cc:1367] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Time spent bootstrapping tablet: real 0.019s user 0.015s sys 0.004s
I20240502 07:59:13.511330 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 24059
I20240502 07:59:13.511763 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:34707
--local_ip_for_outbound_sockets=127.16.36.197
--tserver_master_addrs=127.16.36.254:45791
--webserver_port=35625
--webserver_interface=127.16.36.197
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:13.513526 24080 raft_consensus.cc:348] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:13.514009 24080 raft_consensus.cc:729] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2964136ef5e843c18c95afd711c81b09, State: Initialized, Role: FOLLOWER
I20240502 07:59:13.514564 24080 consensus_queue.cc:260] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:13.515547 24080 ts_tablet_manager.cc:1398] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240502 07:59:13.516599 24188 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:13.516835 24188 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:13.517344 24188 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:13.519312 22753 ts_manager.cc:194] Re-registered known tserver with Master: 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705)
I20240502 07:59:13.520545 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:57565
I20240502 07:59:13.522892 24188 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
W20240502 07:59:13.625530 24192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:13.625773 24192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:13.625818 24192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:13.629061 24192 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:13.629138 24192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:13.629209 24192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:59:13.633026 24192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:34707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=35625
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.24192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:13.633950 24192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:13.635099 24192 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:13.644095 24198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:13.644829 24201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:13.645552 24199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:13.646529 24192 server_base.cc:1034] running on GCE node
I20240502 07:59:13.646943 24192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:13.647591 24192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:13.648768 24192 hybrid_clock.cc:648] HybridClock initialized: now 1714636753648744 us; error 48 us; skew 500 ppm
I20240502 07:59:13.650712 24192 webserver.cc:460] Webserver started at http://127.16.36.197:35625/ using document root <none> and password file <none>
I20240502 07:59:13.651280 24192 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:13.651362 24192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:13.655022 24192 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:13.657173 24207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:13.658382 24192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:13.658596 24192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
format_stamp: "Formatted at 2024-05-02 07:59:05 on dist-test-slave-9sjq"
I20240502 07:59:13.659102 24192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:13.679373 24192 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:13.680074 24192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:13.680238 24192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:13.680858 24192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:13.681792 24192 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:13.681946 24192 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:13.682050 24192 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:13.682075 24192 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:13.691255 24192 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:34707
I20240502 07:59:13.691269 24320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:34707 every 8 connection(s)
I20240502 07:59:13.692306 24192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:13.699482 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 24192
I20240502 07:59:13.699961 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.198:38235
--local_ip_for_outbound_sockets=127.16.36.198
--tserver_master_addrs=127.16.36.254:45791
--webserver_port=40437
--webserver_interface=127.16.36.198
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:13.701912 24321 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:13.702188 24321 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:13.702817 24321 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:13.703913 22753 ts_manager.cc:194] Re-registered known tserver with Master: 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707)
I20240502 07:59:13.704514 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:33997
W20240502 07:59:13.752656 24189 tablet.cc:2369] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:59:13.815564 24324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:13.815855 24324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:13.815905 24324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:13.819183 24324 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:13.819274 24324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:13.819346 24324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.198
I20240502 07:59:13.823171 24324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.198:38235
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.198
--webserver_port=40437
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.24324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.198
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:13.824098 24324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:13.825294 24324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:13.834122 24329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:13.834908 24332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:13.835594 24330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:13.836495 24324 server_base.cc:1034] running on GCE node
I20240502 07:59:13.836983 24324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:13.837615 24324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:13.838810 24324 hybrid_clock.cc:648] HybridClock initialized: now 1714636753838788 us; error 31 us; skew 500 ppm
I20240502 07:59:13.841161 24324 webserver.cc:460] Webserver started at http://127.16.36.198:40437/ using document root <none> and password file <none>
I20240502 07:59:13.841943 24324 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:13.842044 24324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:13.845664 24324 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 07:59:13.847781 24338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:13.848930 24324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:13.849036 24324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "12adf6ad821e4949acc3efc3f6a28404"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:13.849475 24324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:13.864519 24324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:13.865211 24324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:13.865386 24324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:13.865949 24324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:13.867122 24345 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 07:59:13.870376 24324 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 07:59:13.870442 24324 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20240502 07:59:13.870501 24324 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 07:59:13.871990 24324 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 07:59:13.872036 24324 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240502 07:59:13.872318 24345 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap starting.
I20240502 07:59:13.882284 24324 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.198:38235
I20240502 07:59:13.882535 24452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.198:38235 every 8 connection(s)
I20240502 07:59:13.883479 24324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:13.887210 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 24324
I20240502 07:59:13.887637 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.199:34991
--local_ip_for_outbound_sockets=127.16.36.199
--tserver_master_addrs=127.16.36.254:45791
--webserver_port=40147
--webserver_interface=127.16.36.199
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:13.887964 24345 log.cc:824] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:13.893114 24345 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:13.893594 24345 tablet_bootstrap.cc:492] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap complete.
I20240502 07:59:13.894858 24345 ts_tablet_manager.cc:1367] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.003s
I20240502 07:59:13.895367 24453 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:13.895586 24453 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:13.896117 24453 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:13.898021 22753 ts_manager.cc:194] Re-registered known tserver with Master: 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198:38235)
I20240502 07:59:13.898226 24345 raft_consensus.cc:348] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:13.898792 24345 raft_consensus.cc:729] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Initialized, Role: FOLLOWER
I20240502 07:59:13.899011 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.198:33037
I20240502 07:59:13.899291 24345 consensus_queue.cc:260] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:13.900184 24345 ts_tablet_manager.cc:1398] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240502 07:59:13.901633 24453 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
W20240502 07:59:14.004665 24457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:14.004889 24457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:14.004935 24457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:14.008148 24457 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:14.008225 24457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:14.008316 24457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.199
I20240502 07:59:14.012166 24457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.199:34991
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.199
--webserver_port=40147
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.24457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.199
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:14.013163 24457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:14.014336 24457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:14.023511 24464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:14.024317 24466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:14.025019 24463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:14.025844 24457 server_base.cc:1034] running on GCE node
I20240502 07:59:14.026268 24457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:14.026866 24457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:14.028024 24457 hybrid_clock.cc:648] HybridClock initialized: now 1714636754028009 us; error 40 us; skew 500 ppm
I20240502 07:59:14.029991 24457 webserver.cc:460] Webserver started at http://127.16.36.199:40147/ using document root <none> and password file <none>
I20240502 07:59:14.030558 24457 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:14.030642 24457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:14.034507 24457 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.004s
I20240502 07:59:14.036535 24472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:14.037662 24457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:14.037781 24457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e53ee470f3824340a097021b74b80353"
format_stamp: "Formatted at 2024-05-02 07:59:06 on dist-test-slave-9sjq"
I20240502 07:59:14.038240 24457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:14.056594 24457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:14.057335 24457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:14.057516 24457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:14.058099 24457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:14.059041 24457 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:14.059091 24457 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:14.059163 24457 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:14.059188 24457 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:14.067951 24457 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.199:34991
I20240502 07:59:14.067970 24585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.199:34991 every 8 connection(s)
I20240502 07:59:14.069047 24457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:59:14.075953 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 24457
I20240502 07:59:14.076138 16531 test_util.cc:268] Using random seed: -89832982
I20240502 07:59:14.078804 24586 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:14.079095 24586 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:14.079607 24586 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:14.080690 22753 ts_manager.cc:194] Re-registered known tserver with Master: e53ee470f3824340a097021b74b80353 (127.16.36.199:34991)
I20240502 07:59:14.081393 22753 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.199:38123
I20240502 07:59:14.086436 16531 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 07:59:14.086683 16531 test_util.cc:268] Using random seed: -89822440
I20240502 07:59:14.095676 16531 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 07:59:14.095849 16531 test_util.cc:268] Using random seed: -89813274
I20240502 07:59:14.107789 16531 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 07:59:14.107967 16531 test_util.cc:268] Using random seed: -89801155
I20240502 07:59:14.121595 16531 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 07:59:14.121814 16531 test_util.cc:268] Using random seed: -89787311
W20240502 07:59:14.134675 24454 tablet.cc:2369] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:14.137245 16531 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240502 07:59:14.576443 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.580760 24520 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.584764 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:14.592085 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.596657 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:14.614228 24387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.641484 22883 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:14.707137 24321 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
I20240502 07:59:14.957043 24387 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.961488 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.963431 22883 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:14.964795 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:14.968662 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:14.971418 24520 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:14.992764 24766 raft_consensus.cc:482] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:14.993968 24766 raft_consensus.cc:504] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:15.004267 24766 leader_election.cc:288] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198:38235)
I20240502 07:59:15.011643 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:15.021833 24407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbca5be11d04fed987d979b02773974" candidate_uuid: "2964136ef5e843c18c95afd711c81b09" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12adf6ad821e4949acc3efc3f6a28404" is_pre_election: true
I20240502 07:59:15.022218 24407 raft_consensus.cc:2396] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2964136ef5e843c18c95afd711c81b09 for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240502 07:59:15.023351 24075 leader_election.cc:302] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2964136ef5e843c18c95afd711c81b09; no voters: 12adf6ad821e4949acc3efc3f6a28404
I20240502 07:59:15.028256 24766 raft_consensus.cc:2726] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 07:59:15.083921 24586 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
I20240502 07:59:15.183663 22902 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 36, Committed index: 36, Last appended: 1.36, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:15.186141 23165 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 36. Preceding OpId from leader: term: 1 index: 37. (index mismatch)
I20240502 07:59:15.188498 24755 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 37, Last known committed idx: 36, Time since last communication: 0.000s
I20240502 07:59:15.194736 24633 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:15.195785 23164 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Committing config change with OpId 1.37: config changed from index -1 to 37, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:15.199301 22753 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index -1 to 37, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
W20240502 07:59:15.221675 22838 consensus_peers.cc:485] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a -> Peer 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707): Couldn't send request to peer 1f7c07da974747dc9d4d1a2c19c54f02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15fe5361ff8d4e36b6e59b48813f4aaf. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.281450 23163 consensus_queue.cc:237] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:15.285996 22902 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 28. (index mismatch)
I20240502 07:59:15.287246 24613 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240502 07:59:15.289083 24777 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:15.289518 22902 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:15.292570 22758 catalog_manager.cc:5484] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index -1 to 28, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
W20240502 07:59:15.335361 23098 consensus_peers.cc:485] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 -> Peer e53ee470f3824340a097021b74b80353 (127.16.36.199:34991): Couldn't send request to peer e53ee470f3824340a097021b74b80353. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f8ca72508214906a5dcaf459425afe4. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.448930 23163 consensus_queue.cc:237] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:15.455571 23032 raft_consensus.cc:1261] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240502 07:59:15.457046 24613 consensus_queue.cc:1035] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
W20240502 07:59:15.458397 23099 consensus_peers.cc:485] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 -> Peer 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198:38235): Couldn't send request to peer 12adf6ad821e4949acc3efc3f6a28404. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb030dd86b4a4e50a013a0fee3d8dcd2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.462507 24613 raft_consensus.cc:2932] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:15.464323 23033 raft_consensus.cc:2932] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:15.470453 22758 catalog_manager.cc:5484] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index -1 to 46, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:15.475555 24821 raft_consensus.cc:482] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:15.475883 24821 raft_consensus.cc:504] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:15.477087 24821 leader_election.cc:288] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705)
I20240502 07:59:15.483122 24142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbca5be11d04fed987d979b02773974" candidate_uuid: "12adf6ad821e4949acc3efc3f6a28404" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2964136ef5e843c18c95afd711c81b09" is_pre_election: true
I20240502 07:59:15.483407 24142 raft_consensus.cc:2454] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12adf6ad821e4949acc3efc3f6a28404 in term 1.
I20240502 07:59:15.483855 24341 leader_election.cc:302] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12adf6ad821e4949acc3efc3f6a28404, 2964136ef5e843c18c95afd711c81b09; no voters:
I20240502 07:59:15.486178 24821 raft_consensus.cc:2781] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 07:59:15.486337 24821 raft_consensus.cc:482] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:15.486420 24821 raft_consensus.cc:3037] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:15.488718 24821 raft_consensus.cc:504] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:15.489198 24821 leader_election.cc:288] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 2 election: Requested vote from peers 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705)
I20240502 07:59:15.489599 24142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bbca5be11d04fed987d979b02773974" candidate_uuid: "12adf6ad821e4949acc3efc3f6a28404" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2964136ef5e843c18c95afd711c81b09"
I20240502 07:59:15.489735 24142 raft_consensus.cc:3037] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:15.491815 24142 raft_consensus.cc:2454] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12adf6ad821e4949acc3efc3f6a28404 in term 2.
I20240502 07:59:15.492281 24341 leader_election.cc:302] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 12adf6ad821e4949acc3efc3f6a28404, 2964136ef5e843c18c95afd711c81b09; no voters:
I20240502 07:59:15.499832 24821 raft_consensus.cc:2781] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:15.500216 24821 raft_consensus.cc:686] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [term 2 LEADER]: Becoming Leader. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Running, Role: LEADER
I20240502 07:59:15.500701 24821 consensus_queue.cc:237] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:15.503293 22753 catalog_manager.cc:5484] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "12adf6ad821e4949acc3efc3f6a28404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:15.557508 23032 consensus_queue.cc:237] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:15.559376 22901 raft_consensus.cc:1261] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240502 07:59:15.562469 24676 consensus_queue.cc:1035] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240502 07:59:15.566196 23963 raft_consensus.cc:2932] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index -1 to 49, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:15.566907 22902 raft_consensus.cc:2932] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index -1 to 49, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
W20240502 07:59:15.567857 22969 consensus_peers.cc:485] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e -> Peer 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705): Couldn't send request to peer 2964136ef5e843c18c95afd711c81b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37b107413a8b44c2a72967b33ce4f0ee. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.570024 22753 catalog_manager.cc:5484] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index -1 to 49, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:15.621632 22901 consensus_queue.cc:237] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 57, Committed index: 57, Last appended: 1.57, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
W20240502 07:59:15.623768 22838 consensus_peers.cc:485] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a -> Peer 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707): Couldn't send request to peer 1f7c07da974747dc9d4d1a2c19c54f02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 590e0019653d4f16b8d36cfa6f7ab8b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.623842 23163 raft_consensus.cc:1261] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 57. Preceding OpId from leader: term: 1 index: 58. (index mismatch)
I20240502 07:59:15.625036 24633 consensus_queue.cc:1035] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 58, Last known committed idx: 57, Time since last communication: 0.000s
I20240502 07:59:15.627261 24672 raft_consensus.cc:2932] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:15.627681 23163 raft_consensus.cc:2932] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Committing config change with OpId 1.58: config changed from index -1 to 58, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:15.630241 22753 catalog_manager.cc:5484] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index -1 to 58, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:15.720742 24874 ts_tablet_manager.cc:902] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:15.721683 24874 tablet_copy_client.cc:308] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:15.736094 23163 consensus_queue.cc:237] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
W20240502 07:59:15.738703 23098 consensus_peers.cc:485] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 -> Peer e53ee470f3824340a097021b74b80353 (127.16.36.199:34991): Couldn't send request to peer e53ee470f3824340a097021b74b80353. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa8e72ea69b147f4925c7cb6cb8ad24d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.741324 23034 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 1 index: 51. (index mismatch)
I20240502 07:59:15.742564 24671 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20240502 07:59:15.748987 24777 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:15.750433 23033 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.51: config changed from index -1 to 51, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:15.751612 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet 5f8ca72508214906a5dcaf459425afe4 from peer e53ee470f3824340a097021b74b80353 ({username='slave'} at 127.16.36.199:34663)
I20240502 07:59:15.751776 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet 5f8ca72508214906a5dcaf459425afe4 from peer e53ee470f3824340a097021b74b80353 at {username='slave'} at 127.16.36.199:34663: session id = e53ee470f3824340a097021b74b80353-5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:15.753273 23185 tablet_copy_source_session.cc:205] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:15.753844 22758 catalog_manager.cc:5484] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: config changed from index -1 to 51, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:15.759701 24874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8ca72508214906a5dcaf459425afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:15.780511 24881 ts_tablet_manager.cc:902] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Initiating tablet copy from peer 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:15.781808 24874 tablet_copy_client.cc:783] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:15.782033 24874 tablet_copy_client.cc:647] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:15.782989 24881 tablet_copy_client.cc:308] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46399
I20240502 07:59:15.789520 24874 tablet_copy_client.cc:523] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:15.793180 24874 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Bootstrap starting.
I20240502 07:59:15.793467 22923 tablet_copy_service.cc:140] P 5309ff82927d47ff824d67ad15631b0a: Received BeginTabletCopySession request for tablet 15fe5361ff8d4e36b6e59b48813f4aaf from peer 1f7c07da974747dc9d4d1a2c19c54f02 ({username='slave'} at 127.16.36.197:40535)
I20240502 07:59:15.793612 22923 tablet_copy_service.cc:161] P 5309ff82927d47ff824d67ad15631b0a: Beginning new tablet copy session on tablet 15fe5361ff8d4e36b6e59b48813f4aaf from peer 1f7c07da974747dc9d4d1a2c19c54f02 at {username='slave'} at 127.16.36.197:40535: session id = 1f7c07da974747dc9d4d1a2c19c54f02-15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:15.795126 22923 tablet_copy_source_session.cc:205] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:15.799914 24881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fe5361ff8d4e36b6e59b48813f4aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:15.804646 24874 log.cc:824] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:15.824877 24881 tablet_copy_client.cc:783] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:15.825421 24881 tablet_copy_client.cc:647] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:15.829424 24881 tablet_copy_client.cc:523] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:15.833216 24881 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap starting.
I20240502 07:59:15.850829 24881 log.cc:824] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:15.872684 24874 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=669 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:15.873185 24874 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Bootstrap complete.
I20240502 07:59:15.873548 24874 ts_tablet_manager.cc:1367] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Time spent bootstrapping tablet: real 0.081s user 0.055s sys 0.008s
I20240502 07:59:15.876194 24874 raft_consensus.cc:348] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:15.876423 24874 raft_consensus.cc:729] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53ee470f3824340a097021b74b80353, State: Initialized, Role: FOLLOWER
I20240502 07:59:15.876961 24874 consensus_queue.cc:260] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 1.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:15.877939 24874 ts_tablet_manager.cc:1398] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 07:59:15.881714 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session e53ee470f3824340a097021b74b80353-5f8ca72508214906a5dcaf459425afe4 received from {username='slave'} at 127.16.36.199:34663
I20240502 07:59:15.881834 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session e53ee470f3824340a097021b74b80353-5f8ca72508214906a5dcaf459425afe4 on tablet 5f8ca72508214906a5dcaf459425afe4 with peer e53ee470f3824340a097021b74b80353
I20240502 07:59:15.957221 24905 ts_tablet_manager.cc:902] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:15.964236 24905 tablet_copy_client.cc:308] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:15.967546 24142 raft_consensus.cc:1261] T 2bbca5be11d04fed987d979b02773974 P 2964136ef5e843c18c95afd711c81b09 [term 2 FOLLOWER]: Refusing update from remote peer 12adf6ad821e4949acc3efc3f6a28404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 07:59:15.968668 24830 consensus_queue.cc:1035] T 2bbca5be11d04fed987d979b02773974 P 12adf6ad821e4949acc3efc3f6a28404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:15.971657 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet fb030dd86b4a4e50a013a0fee3d8dcd2 from peer 12adf6ad821e4949acc3efc3f6a28404 ({username='slave'} at 127.16.36.198:49093)
I20240502 07:59:15.971778 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet fb030dd86b4a4e50a013a0fee3d8dcd2 from peer 12adf6ad821e4949acc3efc3f6a28404 at {username='slave'} at 127.16.36.198:49093: session id = 12adf6ad821e4949acc3efc3f6a28404-fb030dd86b4a4e50a013a0fee3d8dcd2
I20240502 07:59:15.972738 23185 tablet_copy_source_session.cc:205] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:15.981842 23164 consensus_queue.cc:237] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
W20240502 07:59:15.984542 23099 consensus_peers.cc:485] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 -> Peer 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198:38235): Couldn't send request to peer 12adf6ad821e4949acc3efc3f6a28404. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e91a0e50f7ce40e496812b9d415b2fa4. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:15.987465 23030 raft_consensus.cc:1261] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240502 07:59:15.987612 24905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb030dd86b4a4e50a013a0fee3d8dcd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:15.991079 24905 tablet_copy_client.cc:783] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:15.991143 24777 consensus_queue.cc:1035] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240502 07:59:15.991283 24905 tablet_copy_client.cc:647] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:15.998275 24670 raft_consensus.cc:2932] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:16.002151 23030 raft_consensus.cc:2932] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:16.004643 22753 catalog_manager.cc:5484] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index -1 to 63, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:16.005604 24905 tablet_copy_client.cc:523] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.009389 24905 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap starting.
I20240502 07:59:16.023623 24881 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=62 ignored=0} inserts{seen=693 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:16.024150 24881 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap complete.
I20240502 07:59:16.024586 24881 ts_tablet_manager.cc:1367] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent bootstrapping tablet: real 0.192s user 0.083s sys 0.004s
I20240502 07:59:16.027369 24881 raft_consensus.cc:348] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:16.034802 24881 raft_consensus.cc:729] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.035398 24881 consensus_queue.cc:260] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 62, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:16.036434 24881 ts_tablet_manager.cc:1398] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20240502 07:59:16.046757 22923 tablet_copy_service.cc:342] P 5309ff82927d47ff824d67ad15631b0a: Request end of tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-15fe5361ff8d4e36b6e59b48813f4aaf received from {username='slave'} at 127.16.36.197:40535
I20240502 07:59:16.046869 22923 tablet_copy_service.cc:433] P 5309ff82927d47ff824d67ad15631b0a: ending tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-15fe5361ff8d4e36b6e59b48813f4aaf on tablet 15fe5361ff8d4e36b6e59b48813f4aaf with peer 1f7c07da974747dc9d4d1a2c19c54f02
I20240502 07:59:16.075593 24933 ts_tablet_manager.cc:902] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: Initiating tablet copy from peer 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:16.079453 24933 tablet_copy_client.cc:308] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:38943
I20240502 07:59:16.079478 22901 consensus_queue.cc:237] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.70, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:16.082455 23163 raft_consensus.cc:1261] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 71. (index mismatch)
I20240502 07:59:16.083122 24672 consensus_queue.cc:1035] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 71, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 07:59:16.090009 24862 raft_consensus.cc:2932] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:16.092088 23054 tablet_copy_service.cc:140] P 86a82caa98a74858bb3cb7b6b248f65e: Received BeginTabletCopySession request for tablet 37b107413a8b44c2a72967b33ce4f0ee from peer 2964136ef5e843c18c95afd711c81b09 ({username='slave'} at 127.16.36.196:40091)
I20240502 07:59:16.092262 23054 tablet_copy_service.cc:161] P 86a82caa98a74858bb3cb7b6b248f65e: Beginning new tablet copy session on tablet 37b107413a8b44c2a72967b33ce4f0ee from peer 2964136ef5e843c18c95afd711c81b09 at {username='slave'} at 127.16.36.196:40091: session id = 2964136ef5e843c18c95afd711c81b09-37b107413a8b44c2a72967b33ce4f0ee
I20240502 07:59:16.093890 23054 tablet_copy_source_session.cc:205] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.093890 22753 catalog_manager.cc:5484] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index -1 to 71, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:16.101230 24881 ts_tablet_manager.cc:902] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: Initiating tablet copy from peer 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:16.101694 24881 tablet_copy_client.cc:308] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46399
I20240502 07:59:16.102460 22923 tablet_copy_service.cc:140] P 5309ff82927d47ff824d67ad15631b0a: Received BeginTabletCopySession request for tablet 590e0019653d4f16b8d36cfa6f7ab8b6 from peer 1f7c07da974747dc9d4d1a2c19c54f02 ({username='slave'} at 127.16.36.197:40535)
I20240502 07:59:16.102561 22923 tablet_copy_service.cc:161] P 5309ff82927d47ff824d67ad15631b0a: Beginning new tablet copy session on tablet 590e0019653d4f16b8d36cfa6f7ab8b6 from peer 1f7c07da974747dc9d4d1a2c19c54f02 at {username='slave'} at 127.16.36.197:40535: session id = 1f7c07da974747dc9d4d1a2c19c54f02-590e0019653d4f16b8d36cfa6f7ab8b6
I20240502 07:59:16.103574 22923 tablet_copy_source_session.cc:205] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.104933 24881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 590e0019653d4f16b8d36cfa6f7ab8b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.098140 23163 raft_consensus.cc:2932] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Committing config change with OpId 1.71: config changed from index -1 to 71, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New config: { opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:16.107733 24881 tablet_copy_client.cc:783] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.108119 24881 tablet_copy_client.cc:647] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.110692 24933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37b107413a8b44c2a72967b33ce4f0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.118749 24905 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap replayed 1/1 log segments. Stats: ops{read=63 overwritten=0 applied=63 ignored=0} inserts{seen=735 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.119060 24905 tablet_bootstrap.cc:492] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap complete.
I20240502 07:59:16.119274 24905 ts_tablet_manager.cc:1367] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: Time spent bootstrapping tablet: real 0.110s user 0.069s sys 0.009s
I20240502 07:59:16.119927 24905 raft_consensus.cc:348] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:16.120060 24905 raft_consensus.cc:729] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.120218 24905 consensus_queue.cc:260] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 63, Last appended: 1.63, Last appended by leader: 63, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:16.120544 24905 ts_tablet_manager.cc:1398] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:16.120918 24933 tablet_copy_client.cc:783] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.121191 24933 tablet_copy_client.cc:647] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.125041 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session 12adf6ad821e4949acc3efc3f6a28404-fb030dd86b4a4e50a013a0fee3d8dcd2 received from {username='slave'} at 127.16.36.198:49093
I20240502 07:59:16.125151 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session 12adf6ad821e4949acc3efc3f6a28404-fb030dd86b4a4e50a013a0fee3d8dcd2 on tablet fb030dd86b4a4e50a013a0fee3d8dcd2 with peer 12adf6ad821e4949acc3efc3f6a28404
I20240502 07:59:16.125890 24881 tablet_copy_client.cc:523] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.127426 24881 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap starting.
I20240502 07:59:16.132552 24933 tablet_copy_client.cc:523] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.134138 24933 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: Bootstrap starting.
I20240502 07:59:16.211764 24933 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=59 overwritten=0 applied=59 ignored=0} inserts{seen=744 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.212136 24933 tablet_bootstrap.cc:492] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: Bootstrap complete.
I20240502 07:59:16.212356 24933 ts_tablet_manager.cc:1367] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: Time spent bootstrapping tablet: real 0.078s user 0.062s sys 0.009s
I20240502 07:59:16.213060 24933 raft_consensus.cc:348] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:16.213191 24933 raft_consensus.cc:729] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2964136ef5e843c18c95afd711c81b09, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.213364 24933 consensus_queue.cc:260] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 59, Last appended: 1.59, Last appended by leader: 59, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:16.215611 24933 ts_tablet_manager.cc:1398] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:16.218709 23054 tablet_copy_service.cc:342] P 86a82caa98a74858bb3cb7b6b248f65e: Request end of tablet copy session 2964136ef5e843c18c95afd711c81b09-37b107413a8b44c2a72967b33ce4f0ee received from {username='slave'} at 127.16.36.196:40091
I20240502 07:59:16.218829 23054 tablet_copy_service.cc:433] P 86a82caa98a74858bb3cb7b6b248f65e: ending tablet copy session 2964136ef5e843c18c95afd711c81b09-37b107413a8b44c2a72967b33ce4f0ee on tablet 37b107413a8b44c2a72967b33ce4f0ee with peer 2964136ef5e843c18c95afd711c81b09
I20240502 07:59:16.226074 24275 raft_consensus.cc:1203] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.36->[1.37-1.83] Dedup: 1.63->[1.64-1.83]
W20240502 07:59:16.246264 22838 consensus_peers.cc:485] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a -> Peer 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705): Couldn't send request to peer 2964136ef5e843c18c95afd711c81b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb6626f239b34d5da46b58f4e849145d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:16.270888 24540 raft_consensus.cc:1203] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.27->[1.28-1.53] Dedup: 1.38->[1.39-1.53]
I20240502 07:59:16.279799 24881 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=73 overwritten=0 applied=73 ignored=0} inserts{seen=758 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.280156 24881 tablet_bootstrap.cc:492] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap complete.
I20240502 07:59:16.280411 24881 ts_tablet_manager.cc:1367] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent bootstrapping tablet: real 0.153s user 0.074s sys 0.008s
I20240502 07:59:16.281051 24881 raft_consensus.cc:348] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:16.281176 24881 raft_consensus.cc:729] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.281345 24881 consensus_queue.cc:260] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 73, Last appended: 1.73, Last appended by leader: 73, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 58 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:16.282469 24881 ts_tablet_manager.cc:1398] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:16.283354 22923 tablet_copy_service.cc:342] P 5309ff82927d47ff824d67ad15631b0a: Request end of tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-590e0019653d4f16b8d36cfa6f7ab8b6 received from {username='slave'} at 127.16.36.197:40535
I20240502 07:59:16.283447 22923 tablet_copy_service.cc:433] P 5309ff82927d47ff824d67ad15631b0a: ending tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-590e0019653d4f16b8d36cfa6f7ab8b6 on tablet 590e0019653d4f16b8d36cfa6f7ab8b6 with peer 1f7c07da974747dc9d4d1a2c19c54f02
I20240502 07:59:16.295573 24874 ts_tablet_manager.cc:902] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:16.296203 24874 tablet_copy_client.cc:308] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:16.296742 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet aa8e72ea69b147f4925c7cb6cb8ad24d from peer e53ee470f3824340a097021b74b80353 ({username='slave'} at 127.16.36.199:34663)
I20240502 07:59:16.296859 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet aa8e72ea69b147f4925c7cb6cb8ad24d from peer e53ee470f3824340a097021b74b80353 at {username='slave'} at 127.16.36.199:34663: session id = e53ee470f3824340a097021b74b80353-aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:16.297904 23185 tablet_copy_source_session.cc:205] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.298373 23033 consensus_queue.cc:237] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:16.298852 24874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8e72ea69b147f4925c7cb6cb8ad24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.301190 24874 tablet_copy_client.cc:783] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.301347 24874 tablet_copy_client.cc:647] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.306185 24874 tablet_copy_client.cc:523] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.307308 22901 raft_consensus.cc:1261] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240502 07:59:16.308027 23963 consensus_queue.cc:1035] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 07:59:16.316601 24676 raft_consensus.cc:2932] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index -1 to 86, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:16.319658 22753 catalog_manager.cc:5484] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: config changed from index -1 to 86, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:16.323982 24874 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: Bootstrap starting.
I20240502 07:59:16.324076 22901 raft_consensus.cc:2932] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index -1 to 86, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
W20240502 07:59:16.325551 22969 consensus_peers.cc:485] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e -> Peer 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707): Couldn't send request to peer 1f7c07da974747dc9d4d1a2c19c54f02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 029e2d974d514d6fb63ec3501eb86321. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:16.387137 24407 raft_consensus.cc:1203] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.45->[1.46-1.74] Dedup: 1.63->[1.64-1.74]
I20240502 07:59:16.442657 24905 ts_tablet_manager.cc:902] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:16.443295 24905 tablet_copy_client.cc:308] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:16.443964 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet e91a0e50f7ce40e496812b9d415b2fa4 from peer 12adf6ad821e4949acc3efc3f6a28404 ({username='slave'} at 127.16.36.198:49093)
I20240502 07:59:16.444073 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet e91a0e50f7ce40e496812b9d415b2fa4 from peer 12adf6ad821e4949acc3efc3f6a28404 at {username='slave'} at 127.16.36.198:49093: session id = 12adf6ad821e4949acc3efc3f6a28404-e91a0e50f7ce40e496812b9d415b2fa4
I20240502 07:59:16.445055 23185 tablet_copy_source_session.cc:205] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.446269 24905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91a0e50f7ce40e496812b9d415b2fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.447664 23034 consensus_queue.cc:237] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:16.451468 23163 raft_consensus.cc:1261] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240502 07:59:16.452073 24905 tablet_copy_client.cc:783] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.452265 24905 tablet_copy_client.cc:647] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.454653 24874 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=64 ignored=0} inserts{seen=736 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:16.455184 24874 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: Bootstrap complete.
I20240502 07:59:16.455401 24874 ts_tablet_manager.cc:1367] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: Time spent bootstrapping tablet: real 0.131s user 0.071s sys 0.000s
I20240502 07:59:16.456010 24874 raft_consensus.cc:348] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:16.456633 24874 raft_consensus.cc:729] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53ee470f3824340a097021b74b80353, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.456822 24874 consensus_queue.cc:260] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 51 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:16.457237 24874 ts_tablet_manager.cc:1398] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:16.457691 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session e53ee470f3824340a097021b74b80353-aa8e72ea69b147f4925c7cb6cb8ad24d received from {username='slave'} at 127.16.36.199:34663
I20240502 07:59:16.457772 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session e53ee470f3824340a097021b74b80353-aa8e72ea69b147f4925c7cb6cb8ad24d on tablet aa8e72ea69b147f4925c7cb6cb8ad24d with peer e53ee470f3824340a097021b74b80353
I20240502 07:59:16.463397 24676 consensus_queue.cc:1035] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 83, Time since last communication: 0.000s
I20240502 07:59:16.467736 24688 raft_consensus.cc:2932] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:16.470443 22753 catalog_manager.cc:5484] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: config changed from index -1 to 85, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:16.470512 23164 raft_consensus.cc:2932] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:16.471197 24905 tablet_copy_client.cc:523] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 07:59:16.475346 22967 consensus_peers.cc:485] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e -> Peer e53ee470f3824340a097021b74b80353 (127.16.36.199:34991): Couldn't send request to peer e53ee470f3824340a097021b74b80353. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4ecc981a63c4488aaa9c58a272b6882. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:16.483826 24905 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap starting.
I20240502 07:59:16.537838 24275 raft_consensus.cc:1203] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.57->[1.58-1.88] Dedup: 1.73->[1.74-1.88]
I20240502 07:59:16.626897 24142 raft_consensus.cc:1203] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.48->[1.49-1.68] Dedup: 1.59->[1.60-1.68]
I20240502 07:59:16.661445 24933 ts_tablet_manager.cc:902] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: Initiating tablet copy from peer 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:16.662094 24933 tablet_copy_client.cc:308] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46399
I20240502 07:59:16.672479 23163 consensus_queue.cc:237] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
W20240502 07:59:16.674314 23099 consensus_peers.cc:485] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 -> Peer 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198:38235): Couldn't send request to peer 12adf6ad821e4949acc3efc3f6a28404. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e70ba6d33de4c7eaa320e2dda34ee70. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:16.676283 23032 raft_consensus.cc:1261] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240502 07:59:16.677328 25014 consensus_queue.cc:1035] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240502 07:59:16.689796 22923 tablet_copy_service.cc:140] P 5309ff82927d47ff824d67ad15631b0a: Received BeginTabletCopySession request for tablet eb6626f239b34d5da46b58f4e849145d from peer 2964136ef5e843c18c95afd711c81b09 ({username='slave'} at 127.16.36.196:35703)
I20240502 07:59:16.689925 22923 tablet_copy_service.cc:161] P 5309ff82927d47ff824d67ad15631b0a: Beginning new tablet copy session on tablet eb6626f239b34d5da46b58f4e849145d from peer 2964136ef5e843c18c95afd711c81b09 at {username='slave'} at 127.16.36.196:35703: session id = 2964136ef5e843c18c95afd711c81b09-eb6626f239b34d5da46b58f4e849145d
I20240502 07:59:16.690829 22923 tablet_copy_source_session.cc:205] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.693222 24667 raft_consensus.cc:2932] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:16.696409 24933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb6626f239b34d5da46b58f4e849145d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.698307 22753 catalog_manager.cc:5484] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index -1 to 70, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:16.699132 24933 tablet_copy_client.cc:783] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.699285 24933 tablet_copy_client.cc:647] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.705212 23032 raft_consensus.cc:2932] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, VOTER 12adf6ad821e4949acc3efc3f6a28404 (127.16.36.198) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:16.725620 24933 tablet_copy_client.cc:523] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.727211 24933 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: Bootstrap starting.
I20240502 07:59:16.782286 24905 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap replayed 1/1 log segments. Stats: ops{read=78 overwritten=0 applied=77 ignored=0} inserts{seen=749 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:16.782843 24905 tablet_bootstrap.cc:492] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap complete.
I20240502 07:59:16.783097 24905 ts_tablet_manager.cc:1367] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: Time spent bootstrapping tablet: real 0.299s user 0.083s sys 0.004s
I20240502 07:59:16.783941 24905 raft_consensus.cc:348] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:16.784345 24905 raft_consensus.cc:729] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.784602 24905 consensus_queue.cc:260] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.78, Last appended by leader: 78, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:16.786844 24905 ts_tablet_manager.cc:1398] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:16.789199 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session 12adf6ad821e4949acc3efc3f6a28404-e91a0e50f7ce40e496812b9d415b2fa4 received from {username='slave'} at 127.16.36.198:49093
I20240502 07:59:16.789300 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session 12adf6ad821e4949acc3efc3f6a28404-e91a0e50f7ce40e496812b9d415b2fa4 on tablet e91a0e50f7ce40e496812b9d415b2fa4 with peer 12adf6ad821e4949acc3efc3f6a28404
I20240502 07:59:16.887846 25056 ts_tablet_manager.cc:902] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: Initiating tablet copy from peer 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:16.889298 25056 tablet_copy_client.cc:308] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:38943
I20240502 07:59:16.906175 22901 consensus_queue.cc:237] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:16.906626 24933 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=85 ignored=0} inserts{seen=716 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:16.906947 24933 tablet_bootstrap.cc:492] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: Bootstrap complete.
I20240502 07:59:16.907174 24933 ts_tablet_manager.cc:1367] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: Time spent bootstrapping tablet: real 0.180s user 0.074s sys 0.009s
I20240502 07:59:16.907467 23030 raft_consensus.cc:1261] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240502 07:59:16.907693 24933 raft_consensus.cc:348] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:16.907816 24933 raft_consensus.cc:729] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2964136ef5e843c18c95afd711c81b09, State: Initialized, Role: FOLLOWER
I20240502 07:59:16.908020 24933 consensus_queue.cc:260] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 71 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:16.908085 24862 consensus_queue.cc:1035] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.000s
I20240502 07:59:16.908365 24933 ts_tablet_manager.cc:1398] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:16.909668 24540 raft_consensus.cc:1203] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.50->[1.51-1.81] Dedup: 1.65->[1.66-1.81]
I20240502 07:59:16.912178 22923 tablet_copy_service.cc:342] P 5309ff82927d47ff824d67ad15631b0a: Request end of tablet copy session 2964136ef5e843c18c95afd711c81b09-eb6626f239b34d5da46b58f4e849145d received from {username='slave'} at 127.16.36.196:35703
I20240502 07:59:16.912267 22923 tablet_copy_service.cc:433] P 5309ff82927d47ff824d67ad15631b0a: ending tablet copy session 2964136ef5e843c18c95afd711c81b09-eb6626f239b34d5da46b58f4e849145d on tablet eb6626f239b34d5da46b58f4e849145d with peer 2964136ef5e843c18c95afd711c81b09
W20240502 07:59:16.914209 22838 consensus_peers.cc:485] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a -> Peer 2964136ef5e843c18c95afd711c81b09 (127.16.36.196:42705): Couldn't send request to peer 2964136ef5e843c18c95afd711c81b09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a207b20592e74478beb63c8000b3b1b0. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:16.922324 24672 raft_consensus.cc:2932] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:16.923063 23032 raft_consensus.cc:2932] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index -1 to 113, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:16.930094 23054 tablet_copy_service.cc:140] P 86a82caa98a74858bb3cb7b6b248f65e: Received BeginTabletCopySession request for tablet 029e2d974d514d6fb63ec3501eb86321 from peer 1f7c07da974747dc9d4d1a2c19c54f02 ({username='slave'} at 127.16.36.197:47081)
I20240502 07:59:16.930219 23054 tablet_copy_service.cc:161] P 86a82caa98a74858bb3cb7b6b248f65e: Beginning new tablet copy session on tablet 029e2d974d514d6fb63ec3501eb86321 from peer 1f7c07da974747dc9d4d1a2c19c54f02 at {username='slave'} at 127.16.36.197:47081: session id = 1f7c07da974747dc9d4d1a2c19c54f02-029e2d974d514d6fb63ec3501eb86321
I20240502 07:59:16.930289 22758 catalog_manager.cc:5484] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: config changed from index -1 to 113, VOTER 2964136ef5e843c18c95afd711c81b09 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:16.931185 23054 tablet_copy_source_session.cc:205] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.934063 25056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 029e2d974d514d6fb63ec3501eb86321. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.936823 25056 tablet_copy_client.cc:783] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.936996 25056 tablet_copy_client.cc:647] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.940953 25056 tablet_copy_client.cc:523] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:16.943481 25056 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap starting.
I20240502 07:59:16.947539 24874 ts_tablet_manager.cc:902] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: Initiating tablet copy from peer 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:16.948141 24874 tablet_copy_client.cc:308] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:38943
I20240502 07:59:16.980959 23054 tablet_copy_service.cc:140] P 86a82caa98a74858bb3cb7b6b248f65e: Received BeginTabletCopySession request for tablet f4ecc981a63c4488aaa9c58a272b6882 from peer e53ee470f3824340a097021b74b80353 ({username='slave'} at 127.16.36.199:60847)
I20240502 07:59:16.981081 23054 tablet_copy_service.cc:161] P 86a82caa98a74858bb3cb7b6b248f65e: Beginning new tablet copy session on tablet f4ecc981a63c4488aaa9c58a272b6882 from peer e53ee470f3824340a097021b74b80353 at {username='slave'} at 127.16.36.199:60847: session id = e53ee470f3824340a097021b74b80353-f4ecc981a63c4488aaa9c58a272b6882
I20240502 07:59:16.982031 23054 tablet_copy_source_session.cc:205] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:16.987141 24874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4ecc981a63c4488aaa9c58a272b6882. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:16.989565 24874 tablet_copy_client.cc:783] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:16.989748 24874 tablet_copy_client.cc:647] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:16.996373 24406 raft_consensus.cc:1203] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.62->[1.63-1.90] Dedup: 1.78->[1.79-1.90]
I20240502 07:59:17.012164 24874 tablet_copy_client.cc:523] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:17.018038 24874 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: Bootstrap starting.
I20240502 07:59:17.058008 25056 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=99 overwritten=0 applied=99 ignored=0} inserts{seen=784 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:17.058369 25056 tablet_bootstrap.cc:492] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap complete.
I20240502 07:59:17.058593 25056 ts_tablet_manager.cc:1367] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent bootstrapping tablet: real 0.115s user 0.096s sys 0.000s
I20240502 07:59:17.059083 25056 raft_consensus.cc:348] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:17.059212 25056 raft_consensus.cc:729] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.059381 25056 consensus_queue.cc:260] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.99, Last appended by leader: 99, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:17.059762 25056 ts_tablet_manager.cc:1398] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:17.060797 23054 tablet_copy_service.cc:342] P 86a82caa98a74858bb3cb7b6b248f65e: Request end of tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-029e2d974d514d6fb63ec3501eb86321 received from {username='slave'} at 127.16.36.197:47081
I20240502 07:59:17.060880 23054 tablet_copy_service.cc:433] P 86a82caa98a74858bb3cb7b6b248f65e: ending tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-029e2d974d514d6fb63ec3501eb86321 on tablet 029e2d974d514d6fb63ec3501eb86321 with peer 1f7c07da974747dc9d4d1a2c19c54f02
I20240502 07:59:17.101305 22902 consensus_queue.cc:237] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 77, Committed index: 77, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
W20240502 07:59:17.106299 22838 consensus_peers.cc:485] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a -> Peer 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707): Couldn't send request to peer 1f7c07da974747dc9d4d1a2c19c54f02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bf181a9a54e4a05a354ffbcaf020f4c. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:17.136502 23030 raft_consensus.cc:1261] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240502 07:59:17.137369 24862 consensus_queue.cc:1035] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240502 07:59:17.139057 24862 raft_consensus.cc:2932] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:17.140389 23030 raft_consensus.cc:2932] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:17.143287 22753 catalog_manager.cc:5484] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index -1 to 79, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:17.146356 24874 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=99 ignored=0} inserts{seen=771 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:17.146687 24874 tablet_bootstrap.cc:492] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: Bootstrap complete.
I20240502 07:59:17.147235 24874 ts_tablet_manager.cc:1367] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: Time spent bootstrapping tablet: real 0.129s user 0.091s sys 0.010s
I20240502 07:59:17.149116 24874 raft_consensus.cc:348] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:17.149724 24874 raft_consensus.cc:729] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53ee470f3824340a097021b74b80353, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.150173 24874 consensus_queue.cc:260] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 99, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:17.151075 24874 ts_tablet_manager.cc:1398] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:17.153474 23054 tablet_copy_service.cc:342] P 86a82caa98a74858bb3cb7b6b248f65e: Request end of tablet copy session e53ee470f3824340a097021b74b80353-f4ecc981a63c4488aaa9c58a272b6882 received from {username='slave'} at 127.16.36.199:60847
I20240502 07:59:17.153584 23054 tablet_copy_service.cc:433] P 86a82caa98a74858bb3cb7b6b248f65e: ending tablet copy session e53ee470f3824340a097021b74b80353-f4ecc981a63c4488aaa9c58a272b6882 on tablet f4ecc981a63c4488aaa9c58a272b6882 with peer e53ee470f3824340a097021b74b80353
I20240502 07:59:17.156147 24905 ts_tablet_manager.cc:902] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:17.156837 24905 tablet_copy_client.cc:308] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:17.157390 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet 6e70ba6d33de4c7eaa320e2dda34ee70 from peer 12adf6ad821e4949acc3efc3f6a28404 ({username='slave'} at 127.16.36.198:49093)
I20240502 07:59:17.157516 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet 6e70ba6d33de4c7eaa320e2dda34ee70 from peer 12adf6ad821e4949acc3efc3f6a28404 at {username='slave'} at 127.16.36.198:49093: session id = 12adf6ad821e4949acc3efc3f6a28404-6e70ba6d33de4c7eaa320e2dda34ee70
I20240502 07:59:17.158601 23185 tablet_copy_source_session.cc:205] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:17.159583 24905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e70ba6d33de4c7eaa320e2dda34ee70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.162109 24905 tablet_copy_client.cc:783] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:17.162281 24905 tablet_copy_client.cc:647] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:17.181578 24905 tablet_copy_client.cc:523] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:17.183349 24905 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap starting.
I20240502 07:59:17.208878 24142 raft_consensus.cc:1203] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.70->[1.71-1.92] Dedup: 1.85->[1.86-1.92]
W20240502 07:59:17.244122 23100 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.193:46399, user_credentials={real_user=slave}} blocked reactor thread for 47733.8us
I20240502 07:59:17.288509 24275 raft_consensus.cc:1203] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.85->[1.86-1.102] Dedup: 1.99->[1.100-1.102]
I20240502 07:59:17.316869 24905 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=0} inserts{seen=752 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 07:59:17.317337 24905 tablet_bootstrap.cc:492] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: Bootstrap complete.
I20240502 07:59:17.317539 24905 ts_tablet_manager.cc:1367] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: Time spent bootstrapping tablet: real 0.134s user 0.084s sys 0.003s
I20240502 07:59:17.318001 24905 raft_consensus.cc:348] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:17.318601 24905 raft_consensus.cc:729] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12adf6ad821e4949acc3efc3f6a28404, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.318805 24905 consensus_queue.cc:260] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:17.321338 24905 ts_tablet_manager.cc:1398] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:17.323836 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session 12adf6ad821e4949acc3efc3f6a28404-6e70ba6d33de4c7eaa320e2dda34ee70 received from {username='slave'} at 127.16.36.198:49093
I20240502 07:59:17.323933 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session 12adf6ad821e4949acc3efc3f6a28404-6e70ba6d33de4c7eaa320e2dda34ee70 on tablet 6e70ba6d33de4c7eaa320e2dda34ee70 with peer 12adf6ad821e4949acc3efc3f6a28404
I20240502 07:59:17.340221 22901 consensus_queue.cc:237] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:17.346217 23034 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240502 07:59:17.346573 24862 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240502 07:59:17.353072 24862 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:17.356004 23032 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index -1 to 85, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:17.356485 22758 catalog_manager.cc:5484] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index -1 to 85, VOTER e53ee470f3824340a097021b74b80353 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
W20240502 07:59:17.386521 22836 consensus_peers.cc:485] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a -> Peer e53ee470f3824340a097021b74b80353 (127.16.36.199:34991): Couldn't send request to peer e53ee470f3824340a097021b74b80353. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30773ad41182415ea963b2f59efba756. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:17.395466 24933 ts_tablet_manager.cc:902] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: Initiating tablet copy from peer 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:17.396075 24933 tablet_copy_client.cc:308] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46399
I20240502 07:59:17.396634 22923 tablet_copy_service.cc:140] P 5309ff82927d47ff824d67ad15631b0a: Received BeginTabletCopySession request for tablet a207b20592e74478beb63c8000b3b1b0 from peer 2964136ef5e843c18c95afd711c81b09 ({username='slave'} at 127.16.36.196:35703)
I20240502 07:59:17.396742 22923 tablet_copy_service.cc:161] P 5309ff82927d47ff824d67ad15631b0a: Beginning new tablet copy session on tablet a207b20592e74478beb63c8000b3b1b0 from peer 2964136ef5e843c18c95afd711c81b09 at {username='slave'} at 127.16.36.196:35703: session id = 2964136ef5e843c18c95afd711c81b09-a207b20592e74478beb63c8000b3b1b0
I20240502 07:59:17.397717 22923 tablet_copy_source_session.cc:205] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:17.402696 24933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a207b20592e74478beb63c8000b3b1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.405557 24933 tablet_copy_client.cc:783] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:17.405746 24933 tablet_copy_client.cc:647] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:17.431891 24933 tablet_copy_client.cc:523] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:17.433492 24933 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: Bootstrap starting.
I20240502 07:59:17.460666 24540 raft_consensus.cc:1203] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.84->[1.85-1.111] Dedup: 1.100->[1.101-1.111]
I20240502 07:59:17.477919 23164 consensus_queue.cc:237] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:17.479593 22902 raft_consensus.cc:1261] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240502 07:59:17.480003 24671 consensus_queue.cc:1035] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240502 07:59:17.482012 25014 raft_consensus.cc:2932] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index -1 to 100, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:17.483464 22902 raft_consensus.cc:2932] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index -1 to 100, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:17.484767 22753 catalog_manager.cc:5484] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index -1 to 100, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
W20240502 07:59:17.523663 23100 consensus_peers.cc:485] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 -> Peer 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707): Couldn't send request to peer 1f7c07da974747dc9d4d1a2c19c54f02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c6799e8df114d9c97cf5aafe3233ce2. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:17.660863 24933 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: Bootstrap replayed 1/1 log segments. Stats: ops{read=123 overwritten=0 applied=123 ignored=0} inserts{seen=808 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:17.661191 24933 tablet_bootstrap.cc:492] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: Bootstrap complete.
I20240502 07:59:17.661413 24933 ts_tablet_manager.cc:1367] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: Time spent bootstrapping tablet: real 0.228s user 0.113s sys 0.006s
I20240502 07:59:17.661906 24933 raft_consensus.cc:348] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:17.662556 24933 raft_consensus.cc:729] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2964136ef5e843c18c95afd711c81b09, State: Initialized, Role: FOLLOWER
I20240502 07:59:17.662782 24933 consensus_queue.cc:260] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 123, Last appended: 1.123, Last appended by leader: 123, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:17.663096 24933 ts_tablet_manager.cc:1398] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:17.663868 22923 tablet_copy_service.cc:342] P 5309ff82927d47ff824d67ad15631b0a: Request end of tablet copy session 2964136ef5e843c18c95afd711c81b09-a207b20592e74478beb63c8000b3b1b0 received from {username='slave'} at 127.16.36.196:35703
I20240502 07:59:17.663947 22923 tablet_copy_service.cc:433] P 5309ff82927d47ff824d67ad15631b0a: ending tablet copy session 2964136ef5e843c18c95afd711c81b09-a207b20592e74478beb63c8000b3b1b0 on tablet a207b20592e74478beb63c8000b3b1b0 with peer 2964136ef5e843c18c95afd711c81b09
I20240502 07:59:17.692270 25161 ts_tablet_manager.cc:902] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: Initiating tablet copy from peer 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:17.692812 25161 tablet_copy_client.cc:308] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46399
I20240502 07:59:17.693255 22923 tablet_copy_service.cc:140] P 5309ff82927d47ff824d67ad15631b0a: Received BeginTabletCopySession request for tablet 5bf181a9a54e4a05a354ffbcaf020f4c from peer 1f7c07da974747dc9d4d1a2c19c54f02 ({username='slave'} at 127.16.36.197:40535)
I20240502 07:59:17.693364 22923 tablet_copy_service.cc:161] P 5309ff82927d47ff824d67ad15631b0a: Beginning new tablet copy session on tablet 5bf181a9a54e4a05a354ffbcaf020f4c from peer 1f7c07da974747dc9d4d1a2c19c54f02 at {username='slave'} at 127.16.36.197:40535: session id = 1f7c07da974747dc9d4d1a2c19c54f02-5bf181a9a54e4a05a354ffbcaf020f4c
I20240502 07:59:17.694366 22923 tablet_copy_source_session.cc:205] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:17.700877 25161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf181a9a54e4a05a354ffbcaf020f4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.704860 25161 tablet_copy_client.cc:783] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:17.705052 25161 tablet_copy_client.cc:647] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:17.709939 25161 tablet_copy_client.cc:523] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:17.714821 25161 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap starting.
I20240502 07:59:17.725948 24407 raft_consensus.cc:1203] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.69->[1.70-1.87] Dedup: 1.81->[1.82-1.87]
I20240502 07:59:17.780009 25180 ts_tablet_manager.cc:902] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Initiating tablet copy from peer 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:17.780696 25180 tablet_copy_client.cc:308] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46399
I20240502 07:59:17.789741 22923 tablet_copy_service.cc:140] P 5309ff82927d47ff824d67ad15631b0a: Received BeginTabletCopySession request for tablet 30773ad41182415ea963b2f59efba756 from peer e53ee470f3824340a097021b74b80353 ({username='slave'} at 127.16.36.199:52893)
I20240502 07:59:17.789868 22923 tablet_copy_service.cc:161] P 5309ff82927d47ff824d67ad15631b0a: Beginning new tablet copy session on tablet 30773ad41182415ea963b2f59efba756 from peer e53ee470f3824340a097021b74b80353 at {username='slave'} at 127.16.36.199:52893: session id = e53ee470f3824340a097021b74b80353-30773ad41182415ea963b2f59efba756
I20240502 07:59:17.790853 22923 tablet_copy_source_session.cc:205] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:17.793100 25180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30773ad41182415ea963b2f59efba756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:17.795883 25180 tablet_copy_client.cc:783] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:17.796110 25180 tablet_copy_client.cc:647] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:17.802961 25180 tablet_copy_client.cc:523] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:17.804570 25180 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Bootstrap starting.
I20240502 07:59:17.978677 24142 raft_consensus.cc:1203] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.112->[1.113-1.141] Dedup: 1.123->[1.124-1.141]
I20240502 07:59:17.998351 25180 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Bootstrap replayed 1/1 log segments. Stats: ops{read=97 overwritten=0 applied=97 ignored=0} inserts{seen=728 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:17.998983 25180 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Bootstrap complete.
I20240502 07:59:18.000025 25180 ts_tablet_manager.cc:1367] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Time spent bootstrapping tablet: real 0.196s user 0.091s sys 0.005s
I20240502 07:59:18.002194 25180 raft_consensus.cc:348] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:18.003384 25180 raft_consensus.cc:729] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53ee470f3824340a097021b74b80353, State: Initialized, Role: FOLLOWER
I20240502 07:59:18.003602 25180 consensus_queue.cc:260] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 97, Last appended: 1.97, Last appended by leader: 97, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:18.004026 25180 ts_tablet_manager.cc:1398] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:18.004647 22923 tablet_copy_service.cc:342] P 5309ff82927d47ff824d67ad15631b0a: Request end of tablet copy session e53ee470f3824340a097021b74b80353-30773ad41182415ea963b2f59efba756 received from {username='slave'} at 127.16.36.199:52893
I20240502 07:59:18.004734 22923 tablet_copy_service.cc:433] P 5309ff82927d47ff824d67ad15631b0a: ending tablet copy session e53ee470f3824340a097021b74b80353-30773ad41182415ea963b2f59efba756 on tablet 30773ad41182415ea963b2f59efba756 with peer e53ee470f3824340a097021b74b80353
I20240502 07:59:18.036320 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:18.039542 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:18.068447 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:18.112514 22883 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:18.166638 24520 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:18.195755 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:18.216905 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:18.240391 25219 ts_tablet_manager.cc:902] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:18.248761 24540 raft_consensus.cc:1203] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.84->[1.85-1.108] Dedup: 1.97->[1.98-1.108]
I20240502 07:59:18.257927 25219 tablet_copy_client.cc:308] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:18.317356 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet 1c6799e8df114d9c97cf5aafe3233ce2 from peer 1f7c07da974747dc9d4d1a2c19c54f02 ({username='slave'} at 127.16.36.197:58031)
I20240502 07:59:18.317492 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet 1c6799e8df114d9c97cf5aafe3233ce2 from peer 1f7c07da974747dc9d4d1a2c19c54f02 at {username='slave'} at 127.16.36.197:58031: session id = 1f7c07da974747dc9d4d1a2c19c54f02-1c6799e8df114d9c97cf5aafe3233ce2
I20240502 07:59:18.318454 23185 tablet_copy_source_session.cc:205] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:18.319326 25219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6799e8df114d9c97cf5aafe3233ce2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:18.321585 25219 tablet_copy_client.cc:783] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:18.321764 25219 tablet_copy_client.cc:647] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:18.329895 25219 tablet_copy_client.cc:523] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:18.332779 25219 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap starting.
I20240502 07:59:18.703107 25219 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=112 overwritten=0 applied=112 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:18.703457 25219 tablet_bootstrap.cc:492] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap complete.
I20240502 07:59:18.703686 25219 ts_tablet_manager.cc:1367] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent bootstrapping tablet: real 0.371s user 0.100s sys 0.008s
I20240502 07:59:18.704186 25219 raft_consensus.cc:348] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:18.704324 25219 raft_consensus.cc:729] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Initialized, Role: FOLLOWER
I20240502 07:59:18.704520 25219 consensus_queue.cc:260] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 112, Last appended: 1.112, Last appended by leader: 112, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:18.704922 25219 ts_tablet_manager.cc:1398] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 07:59:18.705834 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-1c6799e8df114d9c97cf5aafe3233ce2 received from {username='slave'} at 127.16.36.197:58031
I20240502 07:59:18.705935 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-1c6799e8df114d9c97cf5aafe3233ce2 on tablet 1c6799e8df114d9c97cf5aafe3233ce2 with peer 1f7c07da974747dc9d4d1a2c19c54f02
I20240502 07:59:18.750622 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:18.751492 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:18.770449 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:18.746809 25161 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=89 ignored=0} inserts{seen=774 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:18.788887 25161 tablet_bootstrap.cc:492] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: Bootstrap complete.
I20240502 07:59:18.790688 25161 ts_tablet_manager.cc:1367] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent bootstrapping tablet: real 1.076s user 0.087s sys 0.012s
I20240502 07:59:18.804808 25161 raft_consensus.cc:348] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:18.805529 25161 raft_consensus.cc:729] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Initialized, Role: FOLLOWER
I20240502 07:59:18.806751 25161 consensus_queue.cc:260] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 89, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:18.808586 25161 ts_tablet_manager.cc:1398] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:18.809653 22923 tablet_copy_service.cc:342] P 5309ff82927d47ff824d67ad15631b0a: Request end of tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-5bf181a9a54e4a05a354ffbcaf020f4c received from {username='slave'} at 127.16.36.197:40535
I20240502 07:59:18.809779 22923 tablet_copy_service.cc:433] P 5309ff82927d47ff824d67ad15631b0a: ending tablet copy session 1f7c07da974747dc9d4d1a2c19c54f02-5bf181a9a54e4a05a354ffbcaf020f4c on tablet 5bf181a9a54e4a05a354ffbcaf020f4c with peer 1f7c07da974747dc9d4d1a2c19c54f02
I20240502 07:59:18.811718 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:18.812438 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:59:18.834986 22969 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.196:42705, user_credentials={real_user=slave}} blocked reactor thread for 52930.1us
I20240502 07:59:18.839360 24275 raft_consensus.cc:1203] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.78->[1.79-1.119] Dedup: 1.89->[1.90-1.119]
I20240502 07:59:18.937894 24520 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:18.961126 22883 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:19.188918 24274 raft_consensus.cc:1203] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.99->[1.100-1.123] Dedup: 1.112->[1.113-1.123]
W20240502 07:59:19.393779 22839 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.195:33213, user_credentials={real_user=slave}} blocked reactor thread for 46268.9us
I20240502 07:59:19.684088 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:19.717037 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:19.720674 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:19.732391 22883 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:19.734637 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:19.761327 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:19.834549 24520 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:20.209877 22902 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30773ad41182415ea963b2f59efba756"
dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37016
I20240502 07:59:20.210103 22902 raft_consensus.cc:595] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Received request to transfer leadership
W20240502 07:59:20.234956 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.251149 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.263531 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.274150 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.282801 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.293205 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.312803 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.336323 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.351569 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.362725 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.380157 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.398419 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.413089 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.428970 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.448539 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.454725 22969 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.193:46399, user_credentials={real_user=slave}} blocked reactor thread for 65139us
I20240502 07:59:20.457043 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:59:20.469744 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
I20240502 07:59:20.483033 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 07:59:20.494357 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.513559 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
I20240502 07:59:20.531301 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:59:20.535243 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
W20240502 07:59:20.559576 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
I20240502 07:59:20.561968 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:20.570907 24520 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:20.572816 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 07:59:20.584612 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
I20240502 07:59:20.599467 22883 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:20.605834 24672 raft_consensus.cc:979] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: : Instructing follower e53ee470f3824340a097021b74b80353 to start an election
I20240502 07:59:20.605957 24672 raft_consensus.cc:1067] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Signalling peer e53ee470f3824340a097021b74b80353 to start an election
W20240502 07:59:20.608106 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Service unavailable: leader transfer in progress
I20240502 07:59:20.609382 24539 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30773ad41182415ea963b2f59efba756"
dest_uuid: "e53ee470f3824340a097021b74b80353"
from {username='slave'} at 127.16.36.193:44187
I20240502 07:59:20.609588 24539 raft_consensus.cc:482] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:20.609730 24539 raft_consensus.cc:3037] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:20.612864 24539 raft_consensus.cc:504] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:20.614023 24539 leader_election.cc:288] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [CANDIDATE]: Term 2 election: Requested vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399), 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:20.614471 22902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30773ad41182415ea963b2f59efba756" candidate_uuid: "e53ee470f3824340a097021b74b80353" candidate_term: 2 candidate_status { last_received { term: 1 index: 155 } } ignore_live_leader: true dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:20.614601 22902 raft_consensus.cc:3032] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:20.614643 22902 raft_consensus.cc:729] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:20.614782 22902 consensus_queue.cc:260] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:20.615007 22902 raft_consensus.cc:3037] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:20.616218 22902 raft_consensus.cc:2454] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53ee470f3824340a097021b74b80353 in term 2.
I20240502 07:59:20.616778 24475 leader_election.cc:302] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5309ff82927d47ff824d67ad15631b0a, e53ee470f3824340a097021b74b80353; no voters:
I20240502 07:59:20.617200 25398 raft_consensus.cc:2781] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:20.617460 25398 raft_consensus.cc:686] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Becoming Leader. State: Replica: e53ee470f3824340a097021b74b80353, State: Running, Role: LEADER
I20240502 07:59:20.619539 25398 consensus_queue.cc:237] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:20.623292 22753 catalog_manager.cc:5484] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 reported cstate change: term changed from 1 to 2, leader changed from 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) to e53ee470f3824340a097021b74b80353 (127.16.36.199). New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:20.631861 23030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30773ad41182415ea963b2f59efba756" candidate_uuid: "e53ee470f3824340a097021b74b80353" candidate_term: 2 candidate_status { last_received { term: 1 index: 155 } } ignore_live_leader: true dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:20.632037 23030 raft_consensus.cc:3037] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:20.633280 23030 raft_consensus.cc:2454] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53ee470f3824340a097021b74b80353 in term 2.
W20240502 07:59:20.634667 22861 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Illegal state: replica 5309ff82927d47ff824d67ad15631b0a is not leader of this config: current role FOLLOWER
W20240502 07:59:20.682396 22993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42350: Illegal state: replica 86a82caa98a74858bb3cb7b6b248f65e is not leader of this config: current role FOLLOWER
W20240502 07:59:20.714576 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36790: Illegal state: replica 5309ff82927d47ff824d67ad15631b0a is not leader of this config: current role FOLLOWER
W20240502 07:59:20.745986 22993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42350: Illegal state: replica 86a82caa98a74858bb3cb7b6b248f65e is not leader of this config: current role FOLLOWER
I20240502 07:59:20.791342 22903 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 2 index: 157. (index mismatch)
I20240502 07:59:20.791844 23030 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 2 index: 157. (index mismatch)
I20240502 07:59:20.792405 25398 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240502 07:59:20.793385 25398 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240502 07:59:20.798830 25411 mvcc.cc:205] Tried to move back new op lower bound from 7023152172194320384 to 7023152171504005120. Current Snapshot: MvccSnapshot[applied={T|T < 7023152172194320384 or (T in {7023152172194320384})}]
I20240502 07:59:21.080520 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:21.085083 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:21.121816 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:21.126682 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:21.131491 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:21.144873 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:21.176890 22883 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:21.512276 23031 consensus_queue.cc:237] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 218, Committed index: 218, Last appended: 1.218, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:21.521896 24538 raft_consensus.cc:1261] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 218. Preceding OpId from leader: term: 1 index: 219. (index mismatch)
I20240502 07:59:21.522513 25345 consensus_queue.cc:1035] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 219, Last known committed idx: 218, Time since last communication: 0.000s
I20240502 07:59:21.525239 24676 raft_consensus.cc:2932] T f4ecc981a63c4488aaa9c58a272b6882 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Committing config change with OpId 1.219: config changed from index 85 to 219, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:21.529613 24539 raft_consensus.cc:2932] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Committing config change with OpId 1.219: config changed from index 85 to 219, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:21.533847 22753 catalog_manager.cc:5484] T f4ecc981a63c4488aaa9c58a272b6882 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 85 to 219, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: 219 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:21.629344 23144 tablet_service.cc:1514] Processing DeleteTablet for tablet f4ecc981a63c4488aaa9c58a272b6882 with delete_type TABLET_DATA_TOMBSTONED (TS d111283808d34d0d8671028dd2d826d5 not found in new config with opid_index 219) from {username='slave'} at 127.0.0.1:52558
I20240502 07:59:21.673883 25488 tablet_replica.cc:330] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: stopping tablet replica
I20240502 07:59:21.683923 25488 raft_consensus.cc:2229] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:21.684298 25488 raft_consensus.cc:2258] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:21.686261 25488 ts_tablet_manager.cc:1875] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:21.690476 25488 ts_tablet_manager.cc:1888] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.218
I20240502 07:59:21.690675 25488 log.cc:1196] T f4ecc981a63c4488aaa9c58a272b6882 P d111283808d34d0d8671028dd2d826d5: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4ecc981a63c4488aaa9c58a272b6882
I20240502 07:59:21.695348 22748 catalog_manager.cc:4830] TS d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213): tablet f4ecc981a63c4488aaa9c58a272b6882 (table rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]) successfully deleted
I20240502 07:59:22.041838 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:22.073002 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:22.097031 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:22.101471 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:22.111951 22883 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 07:59:22.138024 23099 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.16.36.198:38235, user_credentials={real_user=slave}} blocked reactor thread for 66672.2us
I20240502 07:59:22.192951 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:22.202404 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:22.378559 23164 consensus_queue.cc:237] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 251, Committed index: 251, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:22.381376 24405 raft_consensus.cc:1261] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240502 07:59:22.384662 25515 consensus_queue.cc:1035] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 251, Time since last communication: 0.000s
I20240502 07:59:22.389061 25515 raft_consensus.cc:2932] T e91a0e50f7ce40e496812b9d415b2fa4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 63 to 253, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:22.390534 24407 raft_consensus.cc:2932] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 63 to 253, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:22.393675 22752 catalog_manager.cc:5484] T e91a0e50f7ce40e496812b9d415b2fa4 P 12adf6ad821e4949acc3efc3f6a28404 reported cstate change: config changed from index 63 to 253, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:22.529695 23014 tablet_service.cc:1514] Processing DeleteTablet for tablet e91a0e50f7ce40e496812b9d415b2fa4 with delete_type TABLET_DATA_TOMBSTONED (TS 86a82caa98a74858bb3cb7b6b248f65e not found in new config with opid_index 253) from {username='slave'} at 127.0.0.1:50472
I20240502 07:59:22.530202 25570 tablet_replica.cc:330] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: stopping tablet replica
I20240502 07:59:22.530990 25570 raft_consensus.cc:2229] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:22.533741 25570 pending_rounds.cc:62] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Trying to abort 1 pending ops.
I20240502 07:59:22.533859 25570 pending_rounds.cc:68] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Aborting op as it isn't in flight: id { term: 1 index: 252 } timestamp: 7023152178668732416 op_type: WRITE_OP write_request { tablet_id: "e91a0e50f7ce40e496812b9d415b2fa4" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\230\361wL\t\332\305(\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""0745a2a4d971c314" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023152178584403968 authz_token { token_data: "\010\376\221\315\261\006\"3\n\005slave\022*\n 14891407235b40f6aab151a9a138f219\020\001\030\001 \001(\001" signature: "<redacted>""\272~!\315\036\035\274+#H\024q*\344+D\257\"\264\375\325\014p\223\324_\2478\255\333\0076M\267\301\313\351\324\245\260\375\020\213:\272\277\267\275B!\023gp\220\307\003\030\355\235\360pB\213D" signing_key_seq_num: 0 } } request_id { client_id: "f80397d127314d7d9029c76b039e1eeb" seq_no: 681 first_incomplete_seq_no: 681 attempt_no: 0 }
I20240502 07:59:22.534206 25570 raft_consensus.cc:2258] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:22.539407 25570 ts_tablet_manager.cc:1875] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:22.558589 25570 ts_tablet_manager.cc:1888] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.252
I20240502 07:59:22.558899 25570 log.cc:1196] T e91a0e50f7ce40e496812b9d415b2fa4 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/e91a0e50f7ce40e496812b9d415b2fa4
I20240502 07:59:22.562501 22745 catalog_manager.cc:4830] TS 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943): tablet e91a0e50f7ce40e496812b9d415b2fa4 (table rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]) successfully deleted
I20240502 07:59:22.868440 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:22.873687 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:22.891904 24255 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:22.899549 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:22.899638 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:22.913453 22883 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:22.926654 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:23.127490 22903 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf"
dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57068
I20240502 07:59:23.127645 22903 raft_consensus.cc:595] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:23.128834 25540 raft_consensus.cc:979] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: : Instructing follower 1f7c07da974747dc9d4d1a2c19c54f02 to start an election
I20240502 07:59:23.129051 24636 raft_consensus.cc:1067] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Signalling peer 1f7c07da974747dc9d4d1a2c19c54f02 to start an election
I20240502 07:59:23.130306 24272 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf"
dest_uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
from {username='slave'} at 127.16.36.193:37837
I20240502 07:59:23.130563 24272 raft_consensus.cc:482] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:23.130987 24272 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:23.132519 24272 raft_consensus.cc:504] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:23.135483 24272 leader_election.cc:288] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [CANDIDATE]: Term 2 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213), 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:23.136025 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf" candidate_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" candidate_term: 2 candidate_status { last_received { term: 1 index: 280 } } ignore_live_leader: true dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:23.136160 23164 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:23.136502 22901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf" candidate_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" candidate_term: 2 candidate_status { last_received { term: 1 index: 280 } } ignore_live_leader: true dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:23.136611 22901 raft_consensus.cc:3032] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:23.136646 22901 raft_consensus.cc:729] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:23.136766 22901 consensus_queue.cc:260] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 280, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:23.136858 22901 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:23.137389 23164 raft_consensus.cc:2454] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7c07da974747dc9d4d1a2c19c54f02 in term 2.
I20240502 07:59:23.137948 22901 raft_consensus.cc:2454] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7c07da974747dc9d4d1a2c19c54f02 in term 2.
I20240502 07:59:23.138970 24211 leader_election.cc:302] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7c07da974747dc9d4d1a2c19c54f02, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:23.139338 25626 raft_consensus.cc:2781] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:23.139525 25626 raft_consensus.cc:686] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Becoming Leader. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Running, Role: LEADER
I20240502 07:59:23.140897 25626 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 280, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
W20240502 07:59:23.150085 22862 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36812: Illegal state: replica 5309ff82927d47ff824d67ad15631b0a is not leader of this config: current role FOLLOWER
I20240502 07:59:23.144331 22752 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: term changed from 1 to 2, leader changed from 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) to 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197). New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 37 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
W20240502 07:59:23.229373 23124 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:52072: Illegal state: replica d111283808d34d0d8671028dd2d826d5 is not leader of this config: current role FOLLOWER
I20240502 07:59:23.243817 22901 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20240502 07:59:23.244274 25626 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240502 07:59:23.257618 23164 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20240502 07:59:23.258073 25454 mvcc.cc:205] Tried to move back new op lower bound from 7023152182241640448 to 7023152181831360512. Current Snapshot: MvccSnapshot[applied={T|T < 7023152182241640448 or (T in {7023152182241640448})}]
I20240502 07:59:23.262818 25626 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 279, Time since last communication: 0.000s
I20240502 07:59:23.360955 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:23.365840 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:23.366133 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:23.374481 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:23.391728 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:23.404328 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:23.416193 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:23.841799 24273 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 299, Committed index: 299, Last appended: 2.299, Last appended by leader: 280, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:23.842916 23164 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 2 index: 299. Preceding OpId from leader: term: 2 index: 300. (index mismatch)
I20240502 07:59:23.843753 25627 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 300, Last known committed idx: 299, Time since last communication: 0.000s
I20240502 07:59:23.847067 25627 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Committing config change with OpId 2.300: config changed from index 37 to 300, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:23.854518 23163 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Committing config change with OpId 2.300: config changed from index 37 to 300, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:23.857117 22753 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: config changed from index 37 to 300, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 300 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:23.865787 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 15fe5361ff8d4e36b6e59b48813f4aaf with delete_type TABLET_DATA_TOMBSTONED (TS 5309ff82927d47ff824d67ad15631b0a not found in new config with opid_index 300) from {username='slave'} at 127.0.0.1:39416
I20240502 07:59:23.884007 25696 tablet_replica.cc:330] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: stopping tablet replica
I20240502 07:59:23.889212 25696 raft_consensus.cc:2229] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:23.890776 25696 raft_consensus.cc:2258] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:23.895517 25696 ts_tablet_manager.cc:1875] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:23.961776 25696 ts_tablet_manager.cc:1888] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.299
I20240502 07:59:23.962024 25696 log.cc:1196] T 15fe5361ff8d4e36b6e59b48813f4aaf P 5309ff82927d47ff824d67ad15631b0a: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:23.962929 22747 catalog_manager.cc:4830] TS 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399): tablet 15fe5361ff8d4e36b6e59b48813f4aaf (table rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]) successfully deleted
I20240502 07:59:24.354202 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:24.382337 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:24.406533 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:24.407946 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:24.428280 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:24.433307 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:24.433596 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:24.763396 22902 consensus_queue.cc:237] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 268, Committed index: 268, Last appended: 1.268, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:24.765215 24141 raft_consensus.cc:1261] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 268. Preceding OpId from leader: term: 1 index: 269. (index mismatch)
I20240502 07:59:24.765686 24672 consensus_queue.cc:1035] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 269, Last known committed idx: 268, Time since last communication: 0.000s
I20240502 07:59:24.767634 25347 raft_consensus.cc:2932] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.269: config changed from index 71 to 269, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:24.768105 24141 raft_consensus.cc:2932] T eb6626f239b34d5da46b58f4e849145d P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Committing config change with OpId 1.269: config changed from index 71 to 269, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:24.784693 22752 catalog_manager.cc:5484] T eb6626f239b34d5da46b58f4e849145d P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index 71 to 269, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 269 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:24.797663 23144 tablet_service.cc:1514] Processing DeleteTablet for tablet eb6626f239b34d5da46b58f4e849145d with delete_type TABLET_DATA_TOMBSTONED (TS d111283808d34d0d8671028dd2d826d5 not found in new config with opid_index 269) from {username='slave'} at 127.0.0.1:52558
I20240502 07:59:24.798996 25776 tablet_replica.cc:330] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: stopping tablet replica
I20240502 07:59:24.799161 25776 raft_consensus.cc:2229] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:24.799291 25776 raft_consensus.cc:2258] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:24.800048 25776 ts_tablet_manager.cc:1875] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:24.812284 25776 ts_tablet_manager.cc:1888] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.268
I20240502 07:59:24.814072 25776 log.cc:1196] T eb6626f239b34d5da46b58f4e849145d P d111283808d34d0d8671028dd2d826d5: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/eb6626f239b34d5da46b58f4e849145d
I20240502 07:59:24.816617 22748 catalog_manager.cc:4830] TS d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213): tablet eb6626f239b34d5da46b58f4e849145d (table rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]) successfully deleted
I20240502 07:59:25.361682 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:25.418794 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:25.419178 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:25.420670 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:25.455287 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:25.495755 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:25.501055 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:26.048318 23164 consensus_queue.cc:237] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 353, Committed index: 353, Last appended: 1.353, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:26.057615 24405 raft_consensus.cc:1261] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 353. Preceding OpId from leader: term: 1 index: 354. (index mismatch)
I20240502 07:59:26.061416 25515 consensus_queue.cc:1035] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 354, Last known committed idx: 353, Time since last communication: 0.000s
I20240502 07:59:26.071897 24777 raft_consensus.cc:2932] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.354: config changed from index 46 to 354, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:26.079501 22753 catalog_manager.cc:5484] T fb030dd86b4a4e50a013a0fee3d8dcd2 P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index 46 to 354, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:26.090054 24407 raft_consensus.cc:2932] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Committing config change with OpId 1.354: config changed from index 46 to 354, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 354 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:26.141990 23014 tablet_service.cc:1514] Processing DeleteTablet for tablet fb030dd86b4a4e50a013a0fee3d8dcd2 with delete_type TABLET_DATA_TOMBSTONED (TS 86a82caa98a74858bb3cb7b6b248f65e not found in new config with opid_index 354) from {username='slave'} at 127.0.0.1:50472
I20240502 07:59:26.156612 25862 tablet_replica.cc:330] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: stopping tablet replica
I20240502 07:59:26.157207 25862 raft_consensus.cc:2229] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:26.157423 25862 raft_consensus.cc:2258] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:26.268493 25862 ts_tablet_manager.cc:1875] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:26.290925 25862 ts_tablet_manager.cc:1888] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.353
I20240502 07:59:26.291167 25862 log.cc:1196] T fb030dd86b4a4e50a013a0fee3d8dcd2 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/fb030dd86b4a4e50a013a0fee3d8dcd2
I20240502 07:59:26.291913 22745 catalog_manager.cc:4830] TS 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943): tablet fb030dd86b4a4e50a013a0fee3d8dcd2 (table rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]) successfully deleted
I20240502 07:59:26.482182 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:26.501468 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:26.504189 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:26.514503 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:26.516983 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:26.528020 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:26.620445 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:26.990411 23031 consensus_queue.cc:237] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:26.991650 24142 raft_consensus.cc:1261] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240502 07:59:26.992076 25749 consensus_queue.cc:1035] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240502 07:59:26.995997 23963 raft_consensus.cc:2932] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 49 to 280, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:26.996819 24142 raft_consensus.cc:2932] T 37b107413a8b44c2a72967b33ce4f0ee P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 49 to 280, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:26.999306 22759 catalog_manager.cc:5484] T 37b107413a8b44c2a72967b33ce4f0ee P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: config changed from index 49 to 280, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:27.024549 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 37b107413a8b44c2a72967b33ce4f0ee with delete_type TABLET_DATA_TOMBSTONED (TS 5309ff82927d47ff824d67ad15631b0a not found in new config with opid_index 280) from {username='slave'} at 127.0.0.1:39416
I20240502 07:59:27.027323 25932 tablet_replica.cc:330] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: stopping tablet replica
I20240502 07:59:27.027562 25932 raft_consensus.cc:2229] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:27.027772 25932 raft_consensus.cc:2258] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:27.028656 25932 ts_tablet_manager.cc:1875] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:27.032866 25932 ts_tablet_manager.cc:1888] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.279
I20240502 07:59:27.033026 25932 log.cc:1196] T 37b107413a8b44c2a72967b33ce4f0ee P 5309ff82927d47ff824d67ad15631b0a: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/37b107413a8b44c2a72967b33ce4f0ee
I20240502 07:59:27.035166 22747 catalog_manager.cc:4830] TS 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399): tablet 37b107413a8b44c2a72967b33ce4f0ee (table rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]) successfully deleted
I20240502 07:59:27.371893 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:27.389065 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:27.391899 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:27.400195 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:27.405431 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:27.414451 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:27.421021 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:27.759361 22900 consensus_queue.cc:237] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 430, Committed index: 430, Last appended: 1.431, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } }
I20240502 07:59:27.762404 24142 raft_consensus.cc:1261] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 431. Preceding OpId from leader: term: 1 index: 432. (index mismatch)
I20240502 07:59:27.762991 25984 consensus_queue.cc:1035] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 432, Last known committed idx: 430, Time since last communication: 0.000s
I20240502 07:59:27.764884 24672 raft_consensus.cc:2932] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.432: config changed from index 113 to 432, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:27.765786 24140 raft_consensus.cc:2932] T a207b20592e74478beb63c8000b3b1b0 P 2964136ef5e843c18c95afd711c81b09 [term 1 FOLLOWER]: Committing config change with OpId 1.432: config changed from index 113 to 432, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:27.767534 22759 catalog_manager.cc:5484] T a207b20592e74478beb63c8000b3b1b0 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index 113 to 432, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 432 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "2964136ef5e843c18c95afd711c81b09" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 42705 } } }
I20240502 07:59:27.820219 23014 tablet_service.cc:1514] Processing DeleteTablet for tablet a207b20592e74478beb63c8000b3b1b0 with delete_type TABLET_DATA_TOMBSTONED (TS 86a82caa98a74858bb3cb7b6b248f65e not found in new config with opid_index 432) from {username='slave'} at 127.0.0.1:50472
I20240502 07:59:27.823381 26018 tablet_replica.cc:330] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: stopping tablet replica
I20240502 07:59:27.823639 26018 raft_consensus.cc:2229] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:27.823820 26018 pending_rounds.cc:62] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Trying to abort 1 pending ops.
I20240502 07:59:27.823902 26018 pending_rounds.cc:68] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Aborting op as it isn't in flight: id { term: 1 index: 431 } timestamp: 7023152200733020160 op_type: WRITE_OP write_request { tablet_id: "a207b20592e74478beb63c8000b3b1b0" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000g\323S\021\203\3240x\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""8db7806f94a23e8b" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023152200701964288 authz_token { token_data: "\010\376\221\315\261\006\"3\n\005slave\022*\n cb2d66610b344739b19afc07df570028\020\001\030\001 \001(\001" signature: "<redacted>""\346 \346\250k\023\024\347T[~\320q\033>\212\034\n\031\315\357\244\276\035\204a\244#gSQ\001\264\363\337\027[\212\002.\233\372\312:\300\320\251\020\376\325/Y\242<JmkH\320\203\261\350\246\177" signing_key_seq_num: 0 } } request_id { client_id: "2cacc8d52e514b8a943e177a94c08ee2" seq_no: 1236 first_incomplete_seq_no: 1236 attempt_no: 0 }
I20240502 07:59:27.824378 26018 raft_consensus.cc:2258] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:27.825948 26018 ts_tablet_manager.cc:1875] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:27.842018 26018 ts_tablet_manager.cc:1888] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.431
I20240502 07:59:27.844578 26018 log.cc:1196] T a207b20592e74478beb63c8000b3b1b0 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/a207b20592e74478beb63c8000b3b1b0
I20240502 07:59:27.847012 22745 catalog_manager.cc:4830] TS 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943): tablet a207b20592e74478beb63c8000b3b1b0 (table rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]) successfully deleted
I20240502 07:59:28.087244 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.111169 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.154726 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:28.155561 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:28.156625 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.190282 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:28.216881 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.657397 23031 consensus_queue.cc:237] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.381, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:28.658950 24273 raft_consensus.cc:1261] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Refusing update from remote peer 86a82caa98a74858bb3cb7b6b248f65e: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 382. (index mismatch)
I20240502 07:59:28.659379 23963 consensus_queue.cc:1035] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 379, Time since last communication: 0.000s
I20240502 07:59:28.661870 23963 raft_consensus.cc:2932] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 LEADER]: Committing config change with OpId 1.382: config changed from index 86 to 382, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:28.664944 22753 catalog_manager.cc:5484] T 029e2d974d514d6fb63ec3501eb86321 P 86a82caa98a74858bb3cb7b6b248f65e reported cstate change: config changed from index 86 to 382, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "86a82caa98a74858bb3cb7b6b248f65e" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:28.665066 24273 raft_consensus.cc:2932] T 029e2d974d514d6fb63ec3501eb86321 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Committing config change with OpId 1.382: config changed from index 86 to 382, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:28.701817 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 029e2d974d514d6fb63ec3501eb86321 with delete_type TABLET_DATA_TOMBSTONED (TS 5309ff82927d47ff824d67ad15631b0a not found in new config with opid_index 382) from {username='slave'} at 127.0.0.1:39416
I20240502 07:59:28.721460 26095 tablet_replica.cc:330] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: stopping tablet replica
I20240502 07:59:28.721699 26095 raft_consensus.cc:2229] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:28.721834 26095 pending_rounds.cc:62] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Trying to abort 1 pending ops.
I20240502 07:59:28.721941 26095 pending_rounds.cc:68] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Aborting op as it isn't in flight: id { term: 1 index: 381 } timestamp: 7023152204405219328 op_type: WRITE_OP write_request { tablet_id: "029e2d974d514d6fb63ec3501eb86321" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000+\265\224\002\3573S\301\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""ddbce2d637a0bdb5" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023152204381437952 authz_token { token_data: "\010\376\221\315\261\006\"3\n\005slave\022*\n 5a5ae1afbfd34a46a0bc4141f7fdfc0c\020\001\030\001 \001(\001" signature: "<redacted>""Wp\253T\236\332\262\024p\257J\203K!\350WJI\325\344\3333Y\2731\347\325K\361\363\240\216LW\2260\237;\026\244\226^\322\020\311\353\226LN8\270&`\366s\215\371_.c\344\266\311x" signing_key_seq_num: 0 } } request_id { client_id: "4d98fc8466424dca8edb1960dd4d52f5" seq_no: 1128 first_incomplete_seq_no: 1128 attempt_no: 0 }
I20240502 07:59:28.722363 26095 raft_consensus.cc:2258] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:28.723376 26095 ts_tablet_manager.cc:1875] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:28.727170 26095 ts_tablet_manager.cc:1888] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.381
I20240502 07:59:28.728071 26095 log.cc:1196] T 029e2d974d514d6fb63ec3501eb86321 P 5309ff82927d47ff824d67ad15631b0a: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/029e2d974d514d6fb63ec3501eb86321
I20240502 07:59:28.728785 22747 catalog_manager.cc:4830] TS 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399): tablet 029e2d974d514d6fb63ec3501eb86321 (table rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]) successfully deleted
I20240502 07:59:28.863863 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.886483 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.949313 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:28.958199 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:28.958694 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:28.969758 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:28.978866 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:29.124984 22903 consensus_queue.cc:237] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 416, Committed index: 416, Last appended: 1.416, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:29.127238 24273 raft_consensus.cc:1261] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Refusing update from remote peer 5309ff82927d47ff824d67ad15631b0a: Log matching property violated. Preceding OpId in replica: term: 1 index: 416. Preceding OpId from leader: term: 1 index: 417. (index mismatch)
I20240502 07:59:29.127836 25984 consensus_queue.cc:1035] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 417, Last known committed idx: 415, Time since last communication: 0.000s
I20240502 07:59:29.138787 25347 raft_consensus.cc:2932] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Committing config change with OpId 1.417: config changed from index 58 to 417, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:29.142055 22759 catalog_manager.cc:5484] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 5309ff82927d47ff824d67ad15631b0a reported cstate change: config changed from index 58 to 417, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "5309ff82927d47ff824d67ad15631b0a" committed_config { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:29.145902 23144 tablet_service.cc:1514] Processing DeleteTablet for tablet 590e0019653d4f16b8d36cfa6f7ab8b6 with delete_type TABLET_DATA_TOMBSTONED (TS d111283808d34d0d8671028dd2d826d5 not found in new config with opid_index 417) from {username='slave'} at 127.0.0.1:52558
I20240502 07:59:29.146898 24274 raft_consensus.cc:2932] T 590e0019653d4f16b8d36cfa6f7ab8b6 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Committing config change with OpId 1.417: config changed from index 58 to 417, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 417 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:29.149578 26145 tablet_replica.cc:330] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: stopping tablet replica
I20240502 07:59:29.156626 26145 raft_consensus.cc:2229] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:29.157173 26145 raft_consensus.cc:2258] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:29.159018 26145 ts_tablet_manager.cc:1875] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:29.171340 26145 ts_tablet_manager.cc:1888] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.416
I20240502 07:59:29.171746 26145 log.cc:1196] T 590e0019653d4f16b8d36cfa6f7ab8b6 P d111283808d34d0d8671028dd2d826d5: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/590e0019653d4f16b8d36cfa6f7ab8b6
I20240502 07:59:29.173539 22748 catalog_manager.cc:4830] TS d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213): tablet 590e0019653d4f16b8d36cfa6f7ab8b6 (table rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]) successfully deleted
I20240502 07:59:29.446367 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:29.452131 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:29.468765 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:29.501487 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:29.667259 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:29.667699 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:29.691462 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:29.837677 23164 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c6799e8df114d9c97cf5aafe3233ce2"
dest_uuid: "d111283808d34d0d8671028dd2d826d5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52174
I20240502 07:59:29.837877 23164 raft_consensus.cc:595] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:59:29.885453 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.888742 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.891858 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.897265 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.907491 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.914088 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.925123 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.936029 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.948019 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.960726 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.972630 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:29.991994 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.005524 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.022886 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
I20240502 07:59:30.037164 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 07:59:30.043124 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
I20240502 07:59:30.047533 24255 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:30.050822 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:30.053157 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 07:59:30.063575 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
I20240502 07:59:30.067519 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 07:59:30.083884 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.113667 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
I20240502 07:59:30.120302 23144 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:30.131731 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 07:59:30.152407 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.172905 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.197480 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.225780 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.253290 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.279798 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.306318 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
W20240502 07:59:30.363883 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Service unavailable: leader transfer in progress
I20240502 07:59:30.373929 25921 raft_consensus.cc:979] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: : Instructing follower 1f7c07da974747dc9d4d1a2c19c54f02 to start an election
I20240502 07:59:30.374074 25921 raft_consensus.cc:1067] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Signalling peer 1f7c07da974747dc9d4d1a2c19c54f02 to start an election
I20240502 07:59:30.374727 24273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c6799e8df114d9c97cf5aafe3233ce2"
dest_uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
from {username='slave'} at 127.16.36.195:49833
I20240502 07:59:30.374861 24273 raft_consensus.cc:482] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:30.374913 24273 raft_consensus.cc:3037] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:30.376055 24273 raft_consensus.cc:504] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:30.376467 24273 leader_election.cc:288] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [CANDIDATE]: Term 2 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213), 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:30.376811 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6799e8df114d9c97cf5aafe3233ce2" candidate_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" candidate_term: 2 candidate_status { last_received { term: 1 index: 328 } } ignore_live_leader: true dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:30.376982 22900 raft_consensus.cc:3037] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:30.378168 22900 raft_consensus.cc:2454] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7c07da974747dc9d4d1a2c19c54f02 in term 2.
I20240502 07:59:30.378532 24210 leader_election.cc:302] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7c07da974747dc9d4d1a2c19c54f02, 5309ff82927d47ff824d67ad15631b0a; no voters:
I20240502 07:59:30.378741 25626 raft_consensus.cc:2781] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:30.378830 25626 raft_consensus.cc:686] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Becoming Leader. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Running, Role: LEADER
I20240502 07:59:30.378975 25626 consensus_queue.cc:237] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 1.328, Last appended by leader: 328, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:30.379482 22903 raft_consensus.cc:1226] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Rejecting Update request from peer d111283808d34d0d8671028dd2d826d5 for earlier term 1. Current term is 2. Ops: []
I20240502 07:59:30.380806 25610 consensus_queue.cc:1046] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: INVALID_TERM, Last received: 1.328, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 07:59:30.381045 25610 raft_consensus.cc:3032] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:30.381108 25610 raft_consensus.cc:729] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:30.381251 25610 consensus_queue.cc:260] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 328, Committed index: 328, Last appended: 1.328, Last appended by leader: 328, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:30.381410 25610 raft_consensus.cc:3037] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:30.381647 22752 catalog_manager.cc:5484] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: term changed from 1 to 2, leader changed from d111283808d34d0d8671028dd2d826d5 (127.16.36.195) to 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197). New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:30.385509 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6799e8df114d9c97cf5aafe3233ce2" candidate_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" candidate_term: 2 candidate_status { last_received { term: 1 index: 328 } } ignore_live_leader: true dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:30.386765 23164 raft_consensus.cc:2454] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7c07da974747dc9d4d1a2c19c54f02 in term 2.
W20240502 07:59:30.398176 23125 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39478: Illegal state: replica d111283808d34d0d8671028dd2d826d5 is not leader of this config: current role FOLLOWER
I20240502 07:59:30.398514 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 07:59:30.440500 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:57226: Illegal state: replica 5309ff82927d47ff824d67ad15631b0a is not leader of this config: current role FOLLOWER
I20240502 07:59:30.446925 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:30.449410 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:30.450383 23144 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:30.464403 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:30.474784 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:30.477438 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:30.491331 22900 raft_consensus.cc:1261] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
I20240502 07:59:30.491767 25626 consensus_queue.cc:1035] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 07:59:30.494439 25716 mvcc.cc:205] Tried to move back new op lower bound from 7023152211926945792 to 7023152211473686528. Current Snapshot: MvccSnapshot[applied={T|T < 7023152211926945792}]
I20240502 07:59:30.496800 23164 raft_consensus.cc:1261] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 1 index: 328. Preceding OpId from leader: term: 2 index: 330. (index mismatch)
I20240502 07:59:30.497277 25627 consensus_queue.cc:1035] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 329, Last known committed idx: 328, Time since last communication: 0.000s
I20240502 07:59:30.741767 23163 consensus_queue.cc:237] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 447, Committed index: 447, Last appended: 1.447, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } }
I20240502 07:59:30.743475 24406 raft_consensus.cc:1261] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 447. Preceding OpId from leader: term: 1 index: 448. (index mismatch)
I20240502 07:59:30.747231 25991 consensus_queue.cc:1035] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 448, Last known committed idx: 447, Time since last communication: 0.000s
I20240502 07:59:30.752393 25991 raft_consensus.cc:2932] T 6e70ba6d33de4c7eaa320e2dda34ee70 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.448: config changed from index 70 to 448, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:30.754007 24405 raft_consensus.cc:2932] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 [term 1 FOLLOWER]: Committing config change with OpId 1.448: config changed from index 70 to 448, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:30.757787 22759 catalog_manager.cc:5484] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 12adf6ad821e4949acc3efc3f6a28404 reported cstate change: config changed from index 70 to 448, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 448 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "12adf6ad821e4949acc3efc3f6a28404" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 38235 } } }
I20240502 07:59:30.761977 23014 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e70ba6d33de4c7eaa320e2dda34ee70 with delete_type TABLET_DATA_TOMBSTONED (TS 86a82caa98a74858bb3cb7b6b248f65e not found in new config with opid_index 448) from {username='slave'} at 127.0.0.1:50472
I20240502 07:59:30.775810 26313 tablet_replica.cc:330] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: stopping tablet replica
I20240502 07:59:30.777000 26313 raft_consensus.cc:2229] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:30.777483 26313 raft_consensus.cc:2258] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:30.778745 26313 ts_tablet_manager.cc:1875] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:30.790253 26313 ts_tablet_manager.cc:1888] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.447
I20240502 07:59:30.791903 26313 log.cc:1196] T 6e70ba6d33de4c7eaa320e2dda34ee70 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/6e70ba6d33de4c7eaa320e2dda34ee70
I20240502 07:59:30.793197 22745 catalog_manager.cc:4830] TS 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943): tablet 6e70ba6d33de4c7eaa320e2dda34ee70 (table rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]) successfully deleted
I20240502 07:59:31.318686 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:31.329952 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:31.347831 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:31.355046 23144 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:31.359656 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:31.418941 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:31.479568 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:31.638559 23163 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5f8ca72508214906a5dcaf459425afe4"
dest_uuid: "d111283808d34d0d8671028dd2d826d5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52238
I20240502 07:59:31.638715 23163 raft_consensus.cc:595] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Received request to transfer leadership
W20240502 07:59:31.724514 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.728952 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.738456 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.744712 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.754022 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.773497 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.785557 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.798885 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.812572 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.827504 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.839488 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.853617 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.872327 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.889163 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.914379 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
W20240502 07:59:31.931658 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
I20240502 07:59:31.944682 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:31.949542 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 07:59:31.950485 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
I20240502 07:59:31.960969 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 07:59:31.968818 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
I20240502 07:59:31.970952 24520 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:31.977404 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:31.978569 23144 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 07:59:31.978897 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 07:59:31.991642 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
I20240502 07:59:31.999776 25921 raft_consensus.cc:979] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: : Instructing follower e53ee470f3824340a097021b74b80353 to start an election
I20240502 07:59:31.999891 25921 raft_consensus.cc:1067] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Signalling peer e53ee470f3824340a097021b74b80353 to start an election
I20240502 07:59:32.000635 24538 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f8ca72508214906a5dcaf459425afe4"
dest_uuid: "e53ee470f3824340a097021b74b80353"
from {username='slave'} at 127.16.36.195:35077
I20240502 07:59:32.000861 24538 raft_consensus.cc:482] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:32.001055 24538 raft_consensus.cc:3037] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:32.002918 24538 raft_consensus.cc:504] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
W20240502 07:59:32.013975 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Service unavailable: leader transfer in progress
I20240502 07:59:32.017066 24538 leader_election.cc:288] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [CANDIDATE]: Term 2 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213), 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399)
I20240502 07:59:32.018481 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8ca72508214906a5dcaf459425afe4" candidate_uuid: "e53ee470f3824340a097021b74b80353" candidate_term: 2 candidate_status { last_received { term: 1 index: 476 } } ignore_live_leader: true dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:32.018532 22900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f8ca72508214906a5dcaf459425afe4" candidate_uuid: "e53ee470f3824340a097021b74b80353" candidate_term: 2 candidate_status { last_received { term: 1 index: 476 } } ignore_live_leader: true dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:32.018656 22900 raft_consensus.cc:3037] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:32.018749 23164 raft_consensus.cc:3032] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:32.018795 23164 raft_consensus.cc:729] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:32.018937 23164 consensus_queue.cc:260] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 476, Committed index: 476, Last appended: 1.476, Last appended by leader: 476, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:32.019086 23164 raft_consensus.cc:3037] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:32.019837 22900 raft_consensus.cc:2454] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53ee470f3824340a097021b74b80353 in term 2.
I20240502 07:59:32.020114 23164 raft_consensus.cc:2454] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53ee470f3824340a097021b74b80353 in term 2.
I20240502 07:59:32.020442 24476 leader_election.cc:302] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d111283808d34d0d8671028dd2d826d5, e53ee470f3824340a097021b74b80353; no voters:
I20240502 07:59:32.020785 26025 raft_consensus.cc:2781] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:32.020884 26025 raft_consensus.cc:686] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Becoming Leader. State: Replica: e53ee470f3824340a097021b74b80353, State: Running, Role: LEADER
I20240502 07:59:32.021023 26025 consensus_queue.cc:237] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 476, Committed index: 476, Last appended: 1.476, Last appended by leader: 476, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:32.023027 22752 catalog_manager.cc:5484] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 reported cstate change: term changed from 1 to 2, leader changed from d111283808d34d0d8671028dd2d826d5 (127.16.36.195) to e53ee470f3824340a097021b74b80353 (127.16.36.199). New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
W20240502 07:59:32.035300 23123 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39476: Illegal state: replica d111283808d34d0d8671028dd2d826d5 is not leader of this config: current role FOLLOWER
W20240502 07:59:32.069514 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39328: Illegal state: replica 5309ff82927d47ff824d67ad15631b0a is not leader of this config: current role FOLLOWER
I20240502 07:59:32.119678 22902 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 1 index: 476. Preceding OpId from leader: term: 2 index: 478. (index mismatch)
I20240502 07:59:32.120543 26240 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I20240502 07:59:32.124713 23163 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 1 index: 476. Preceding OpId from leader: term: 2 index: 478. (index mismatch)
I20240502 07:59:32.125789 26025 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 477, Last known committed idx: 476, Time since last communication: 0.000s
I20240502 07:59:32.362651 24539 consensus_queue.cc:237] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 487, Committed index: 487, Last appended: 2.487, Last appended by leader: 476, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:32.365124 22900 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 487. Preceding OpId from leader: term: 2 index: 488. (index mismatch)
I20240502 07:59:32.365655 26025 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 488, Last known committed idx: 487, Time since last communication: 0.000s
I20240502 07:59:32.367873 26025 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.488: config changed from index 28 to 488, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:32.375939 22752 catalog_manager.cc:5484] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 28 to 488, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:32.377768 22902 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Committing config change with OpId 2.488: config changed from index 28 to 488, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 488 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:32.379422 23144 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8ca72508214906a5dcaf459425afe4 with delete_type TABLET_DATA_TOMBSTONED (TS d111283808d34d0d8671028dd2d826d5 not found in new config with opid_index 488) from {username='slave'} at 127.0.0.1:52558
I20240502 07:59:32.386837 26450 tablet_replica.cc:330] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: stopping tablet replica
I20240502 07:59:32.387113 26450 raft_consensus.cc:2229] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:32.387856 26450 raft_consensus.cc:2258] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:32.393646 26450 ts_tablet_manager.cc:1875] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:32.396234 26450 ts_tablet_manager.cc:1888] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.487
I20240502 07:59:32.396399 26450 log.cc:1196] T 5f8ca72508214906a5dcaf459425afe4 P d111283808d34d0d8671028dd2d826d5: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:32.397768 22748 catalog_manager.cc:4830] TS d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213): tablet 5f8ca72508214906a5dcaf459425afe4 (table rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]) successfully deleted
I20240502 07:59:32.667896 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:32.675915 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:32.681316 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:32.682149 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:32.693281 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:32.711007 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:32.716583 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:32.877907 23164 consensus_queue.cc:237] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 411, Committed index: 411, Last appended: 1.411, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:32.882055 24538 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 411. Preceding OpId from leader: term: 1 index: 412. (index mismatch)
I20240502 07:59:32.882622 25991 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 412, Last known committed idx: 411, Time since last communication: 0.000s
I20240502 07:59:32.885525 25991 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.412: config changed from index 51 to 412, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:32.889586 24538 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Committing config change with OpId 1.412: config changed from index 51 to 412, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:32.890473 22752 catalog_manager.cc:5484] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index 51 to 412, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 412 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:32.943176 23014 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8e72ea69b147f4925c7cb6cb8ad24d with delete_type TABLET_DATA_TOMBSTONED (TS 86a82caa98a74858bb3cb7b6b248f65e not found in new config with opid_index 412) from {username='slave'} at 127.0.0.1:50472
I20240502 07:59:32.950060 26519 tablet_replica.cc:330] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: stopping tablet replica
I20240502 07:59:32.950549 26519 raft_consensus.cc:2229] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:32.950735 26519 raft_consensus.cc:2258] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:32.953485 26519 ts_tablet_manager.cc:1875] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:32.956923 26519 ts_tablet_manager.cc:1888] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.411
I20240502 07:59:32.957024 26519 log.cc:1196] T aa8e72ea69b147f4925c7cb6cb8ad24d P 86a82caa98a74858bb3cb7b6b248f65e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:32.957667 22745 catalog_manager.cc:4830] TS 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943): tablet aa8e72ea69b147f4925c7cb6cb8ad24d (table rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]) successfully deleted
I20240502 07:59:33.351897 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:33.403681 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:33.406241 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.412341 22883 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:33.412603 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:33.447707 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:33.503193 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:33.606649 22903 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bf181a9a54e4a05a354ffbcaf020f4c"
dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39368
I20240502 07:59:33.606802 22903 raft_consensus.cc:595] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:33.607975 26365 raft_consensus.cc:979] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: : Instructing follower 1f7c07da974747dc9d4d1a2c19c54f02 to start an election
I20240502 07:59:33.608088 26365 raft_consensus.cc:1067] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Signalling peer 1f7c07da974747dc9d4d1a2c19c54f02 to start an election
I20240502 07:59:33.614471 24273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bf181a9a54e4a05a354ffbcaf020f4c"
dest_uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
from {username='slave'} at 127.16.36.193:37837
I20240502 07:59:33.614647 24273 raft_consensus.cc:482] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:33.614723 24273 raft_consensus.cc:3037] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:33.615837 24273 raft_consensus.cc:504] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:33.617064 22902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf181a9a54e4a05a354ffbcaf020f4c" candidate_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" candidate_term: 2 candidate_status { last_received { term: 1 index: 513 } } ignore_live_leader: true dest_uuid: "5309ff82927d47ff824d67ad15631b0a"
I20240502 07:59:33.617198 22902 raft_consensus.cc:3032] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:33.617244 22902 raft_consensus.cc:729] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5309ff82927d47ff824d67ad15631b0a, State: Running, Role: LEADER
I20240502 07:59:33.617378 22902 consensus_queue.cc:260] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 513, Committed index: 513, Last appended: 1.513, Last appended by leader: 513, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:33.617549 22902 raft_consensus.cc:3037] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:33.619416 22902 raft_consensus.cc:2454] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7c07da974747dc9d4d1a2c19c54f02 in term 2.
I20240502 07:59:33.621553 24273 leader_election.cc:288] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [CANDIDATE]: Term 2 election: Requested vote from peers 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399), 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943)
I20240502 07:59:33.621654 23031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf181a9a54e4a05a354ffbcaf020f4c" candidate_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" candidate_term: 2 candidate_status { last_received { term: 1 index: 513 } } ignore_live_leader: true dest_uuid: "86a82caa98a74858bb3cb7b6b248f65e"
I20240502 07:59:33.621786 23031 raft_consensus.cc:3037] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:33.622068 24210 leader_election.cc:302] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7c07da974747dc9d4d1a2c19c54f02, 5309ff82927d47ff824d67ad15631b0a; no voters:
I20240502 07:59:33.622251 26415 raft_consensus.cc:2781] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:33.622330 26415 raft_consensus.cc:686] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Becoming Leader. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Running, Role: LEADER
I20240502 07:59:33.622457 26415 consensus_queue.cc:237] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 513, Committed index: 513, Last appended: 1.513, Last appended by leader: 513, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:33.622910 23031 raft_consensus.cc:2454] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f7c07da974747dc9d4d1a2c19c54f02 in term 2.
I20240502 07:59:33.625308 22761 catalog_manager.cc:5484] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: term changed from 1 to 2, leader changed from 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) to 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197). New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
W20240502 07:59:33.640318 22860 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:36802: Illegal state: replica 5309ff82927d47ff824d67ad15631b0a is not leader of this config: current role FOLLOWER
W20240502 07:59:33.678455 22993 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37546: Illegal state: replica 86a82caa98a74858bb3cb7b6b248f65e is not leader of this config: current role FOLLOWER
I20240502 07:59:33.727819 22903 raft_consensus.cc:1261] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 1 index: 513. Preceding OpId from leader: term: 2 index: 515. (index mismatch)
I20240502 07:59:33.728328 26415 consensus_queue.cc:1035] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 514, Last known committed idx: 513, Time since last communication: 0.000s
I20240502 07:59:33.729389 23032 raft_consensus.cc:1261] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 1 index: 513. Preceding OpId from leader: term: 2 index: 515. (index mismatch)
I20240502 07:59:33.730788 26415 consensus_queue.cc:1035] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 514, Last known committed idx: 513, Time since last communication: 0.000s
I20240502 07:59:33.739756 26304 mvcc.cc:205] Tried to move back new op lower bound from 7023152225184993280 to 7023152224759037952. Current Snapshot: MvccSnapshot[applied={T|T < 7023152225184993280}]
I20240502 07:59:33.900031 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:33.900343 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:33.932005 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:33.933521 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:33.951143 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:33.952731 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:33.963341 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:34.212899 24273 consensus_queue.cc:237] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 533, Committed index: 533, Last appended: 2.533, Last appended by leader: 513, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:34.216215 23032 raft_consensus.cc:1261] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 2 index: 533. Preceding OpId from leader: term: 2 index: 534. (index mismatch)
I20240502 07:59:34.217043 25626 consensus_queue.cc:1035] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 534, Last known committed idx: 533, Time since last communication: 0.000s
I20240502 07:59:34.220831 25626 raft_consensus.cc:2932] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Committing config change with OpId 2.534: config changed from index 79 to 534, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:34.221120 23031 raft_consensus.cc:2932] T 5bf181a9a54e4a05a354ffbcaf020f4c P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Committing config change with OpId 2.534: config changed from index 79 to 534, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:34.224671 22753 catalog_manager.cc:5484] T 5bf181a9a54e4a05a354ffbcaf020f4c P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: config changed from index 79 to 534, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:34.263180 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf181a9a54e4a05a354ffbcaf020f4c with delete_type TABLET_DATA_TOMBSTONED (TS 5309ff82927d47ff824d67ad15631b0a not found in new config with opid_index 534) from {username='slave'} at 127.0.0.1:39416
I20240502 07:59:34.275355 26661 tablet_replica.cc:330] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: stopping tablet replica
I20240502 07:59:34.275611 26661 raft_consensus.cc:2229] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:34.275791 26661 raft_consensus.cc:2258] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:34.278561 26661 ts_tablet_manager.cc:1875] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:34.281560 26661 ts_tablet_manager.cc:1888] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.533
I20240502 07:59:34.281747 26661 log.cc:1196] T 5bf181a9a54e4a05a354ffbcaf020f4c P 5309ff82927d47ff824d67ad15631b0a: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/5bf181a9a54e4a05a354ffbcaf020f4c
I20240502 07:59:34.289741 22747 catalog_manager.cc:4830] TS 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399): tablet 5bf181a9a54e4a05a354ffbcaf020f4c (table rebalance_test_table_2 [id=14891407235b40f6aab151a9a138f219]) successfully deleted
I20240502 07:59:34.549218 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:34.572080 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:34.592552 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:34.595175 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:34.595469 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:34.623023 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:34.676398 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:34.823680 24538 consensus_queue.cc:237] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 528, Committed index: 528, Last appended: 2.528, Last appended by leader: 155, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } }
I20240502 07:59:34.824942 23031 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 528. Preceding OpId from leader: term: 2 index: 529. (index mismatch)
I20240502 07:59:34.826524 26573 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 529, Last known committed idx: 528, Time since last communication: 0.000s
I20240502 07:59:34.829272 26573 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.529: config changed from index 85 to 529, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:34.829602 23031 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Committing config change with OpId 2.529: config changed from index 85 to 529, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:34.832149 22753 catalog_manager.cc:5484] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 85 to 529, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 529 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } }
I20240502 07:59:34.839609 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 30773ad41182415ea963b2f59efba756 with delete_type TABLET_DATA_TOMBSTONED (TS 5309ff82927d47ff824d67ad15631b0a not found in new config with opid_index 529) from {username='slave'} at 127.0.0.1:39416
I20240502 07:59:34.839944 26719 tablet_replica.cc:330] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: stopping tablet replica
I20240502 07:59:34.840283 26719 raft_consensus.cc:2229] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:34.840502 26719 raft_consensus.cc:2258] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:34.841377 26719 ts_tablet_manager.cc:1875] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:34.855664 26719 ts_tablet_manager.cc:1888] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.528
I20240502 07:59:34.855862 26719 log.cc:1196] T 30773ad41182415ea963b2f59efba756 P 5309ff82927d47ff824d67ad15631b0a: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/30773ad41182415ea963b2f59efba756
I20240502 07:59:34.858115 22747 catalog_manager.cc:4830] TS 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399): tablet 30773ad41182415ea963b2f59efba756 (table rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]) successfully deleted
I20240502 07:59:35.397539 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:35.427848 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:35.428220 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:35.430466 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:35.433387 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:35.444859 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:35.584303 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:35.693691 24273 consensus_queue.cc:237] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 474, Committed index: 474, Last appended: 2.474, Last appended by leader: 328, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } }
I20240502 07:59:35.694722 22900 raft_consensus.cc:1261] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 2 index: 474. Preceding OpId from leader: term: 2 index: 475. (index mismatch)
I20240502 07:59:35.695765 25626 consensus_queue.cc:1035] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 475, Last known committed idx: 474, Time since last communication: 0.000s
I20240502 07:59:35.698009 26415 raft_consensus.cc:2932] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Committing config change with OpId 2.475: config changed from index 100 to 475, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:35.701581 22757 catalog_manager.cc:5484] T 1c6799e8df114d9c97cf5aafe3233ce2 P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: config changed from index 100 to 475, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:35.702551 22900 raft_consensus.cc:2932] T 1c6799e8df114d9c97cf5aafe3233ce2 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Committing config change with OpId 2.475: config changed from index 100 to 475, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 475 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } }
I20240502 07:59:35.737864 23144 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6799e8df114d9c97cf5aafe3233ce2 with delete_type TABLET_DATA_TOMBSTONED (TS d111283808d34d0d8671028dd2d826d5 not found in new config with opid_index 475) from {username='slave'} at 127.0.0.1:52558
I20240502 07:59:35.738528 26838 tablet_replica.cc:330] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: stopping tablet replica
I20240502 07:59:35.738737 26838 raft_consensus.cc:2229] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:35.738865 26838 raft_consensus.cc:2258] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:35.740199 26838 ts_tablet_manager.cc:1875] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:35.747602 26838 ts_tablet_manager.cc:1888] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.474
I20240502 07:59:35.747695 26838 log.cc:1196] T 1c6799e8df114d9c97cf5aafe3233ce2 P d111283808d34d0d8671028dd2d826d5: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/1c6799e8df114d9c97cf5aafe3233ce2
I20240502 07:59:35.752415 22748 catalog_manager.cc:4830] TS d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213): tablet 1c6799e8df114d9c97cf5aafe3233ce2 (table rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]) successfully deleted
I20240502 07:59:36.221422 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:36.232661 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:36.243192 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:36.275463 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:36.278996 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:36.290086 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:36.317008 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:36.664048 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:36.666812 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:36.670010 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:36.680214 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:36.680773 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:36.695894 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:36.730051 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:37.387488 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:37.399133 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:37.399964 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:37.404445 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:37.418959 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:37.455802 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:37.466826 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:37.814119 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:37.823110 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:37.823877 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:37.825112 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:37.838059 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:37.858276 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:37.875044 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:38.108803 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:38.109133 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:38.114652 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:38.114936 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:38.126232 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:38.128994 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:38.140187 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5229ea5962df43b293bb46e794006e25 | 127.16.36.254:45791 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.212:34767 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
12adf6ad821e4949acc3efc3f6a28404 | 127.16.36.198:38235 | HEALTHY | <none> | 1 | 0
1f7c07da974747dc9d4d1a2c19c54f02 | 127.16.36.197:34707 | HEALTHY | <none> | 3 | 0
2964136ef5e843c18c95afd711c81b09 | 127.16.36.196:42705 | HEALTHY | <none> | 0 | 0
5309ff82927d47ff824d67ad15631b0a | 127.16.36.193:46399 | HEALTHY | <none> | 3 | 0
86a82caa98a74858bb3cb7b6b248f65e | 127.16.36.194:38943 | HEALTHY | <none> | 3 | 0
d111283808d34d0d8671028dd2d826d5 | 127.16.36.195:33213 | HEALTHY | <none> | 4 | 0
e53ee470f3824340a097021b74b80353 | 127.16.36.199:34991 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.16.36.193 | experimental | 127.16.36.193:46399
local_ip_for_outbound_sockets | 127.16.36.194 | experimental | 127.16.36.194:38943
local_ip_for_outbound_sockets | 127.16.36.195 | experimental | 127.16.36.195:33213
local_ip_for_outbound_sockets | 127.16.36.196 | experimental | 127.16.36.196:42705
local_ip_for_outbound_sockets | 127.16.36.197 | experimental | 127.16.36.197:34707
local_ip_for_outbound_sockets | 127.16.36.198 | experimental | 127.16.36.198:38235
local_ip_for_outbound_sockets | 127.16.36.199 | experimental | 127.16.36.199:34991
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.36.193:46399
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.36.194:38943
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.36.195:33213
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.36.196:42705
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.36.197:34707
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.36.198:38235
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.36.199:34991
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.212:34767 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 07:59:38.169765 16531 log_verifier.cc:126] Checking tablet 029e2d974d514d6fb63ec3501eb86321
I20240502 07:59:38.276973 16531 log_verifier.cc:177] Verified matching terms for 662 ops in tablet 029e2d974d514d6fb63ec3501eb86321
I20240502 07:59:38.277206 16531 log_verifier.cc:126] Checking tablet 15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:38.385874 16531 log_verifier.cc:177] Verified matching terms for 697 ops in tablet 15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:38.386155 16531 log_verifier.cc:126] Checking tablet 1c6799e8df114d9c97cf5aafe3233ce2
I20240502 07:59:38.474429 16531 log_verifier.cc:177] Verified matching terms for 543 ops in tablet 1c6799e8df114d9c97cf5aafe3233ce2
I20240502 07:59:38.474603 16531 log_verifier.cc:126] Checking tablet 2bbca5be11d04fed987d979b02773974
I20240502 07:59:38.489092 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2bbca5be11d04fed987d979b02773974
I20240502 07:59:38.489161 16531 log_verifier.cc:126] Checking tablet 30773ad41182415ea963b2f59efba756
I20240502 07:59:38.591132 16531 log_verifier.cc:177] Verified matching terms for 617 ops in tablet 30773ad41182415ea963b2f59efba756
I20240502 07:59:38.591313 16531 log_verifier.cc:126] Checking tablet 37b107413a8b44c2a72967b33ce4f0ee
I20240502 07:59:38.688786 16531 log_verifier.cc:177] Verified matching terms for 599 ops in tablet 37b107413a8b44c2a72967b33ce4f0ee
I20240502 07:59:38.688966 16531 log_verifier.cc:126] Checking tablet 590e0019653d4f16b8d36cfa6f7ab8b6
I20240502 07:59:38.793308 16531 log_verifier.cc:177] Verified matching terms for 657 ops in tablet 590e0019653d4f16b8d36cfa6f7ab8b6
I20240502 07:59:38.793529 16531 log_verifier.cc:126] Checking tablet 5bf181a9a54e4a05a354ffbcaf020f4c
I20240502 07:59:38.900784 16531 log_verifier.cc:177] Verified matching terms for 664 ops in tablet 5bf181a9a54e4a05a354ffbcaf020f4c
I20240502 07:59:38.901047 16531 log_verifier.cc:126] Checking tablet 5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:39.012329 16531 log_verifier.cc:177] Verified matching terms for 675 ops in tablet 5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:39.012588 16531 log_verifier.cc:126] Checking tablet 6e70ba6d33de4c7eaa320e2dda34ee70
I20240502 07:59:39.123800 16531 log_verifier.cc:177] Verified matching terms for 668 ops in tablet 6e70ba6d33de4c7eaa320e2dda34ee70
I20240502 07:59:39.124027 16531 log_verifier.cc:126] Checking tablet a207b20592e74478beb63c8000b3b1b0
I20240502 07:59:39.237473 16531 log_verifier.cc:177] Verified matching terms for 705 ops in tablet a207b20592e74478beb63c8000b3b1b0
I20240502 07:59:39.237704 16531 log_verifier.cc:126] Checking tablet aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:39.331099 16531 log_verifier.cc:177] Verified matching terms for 587 ops in tablet aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:39.331300 16531 log_verifier.cc:126] Checking tablet e91a0e50f7ce40e496812b9d415b2fa4
I20240502 07:59:39.437909 16531 log_verifier.cc:177] Verified matching terms for 667 ops in tablet e91a0e50f7ce40e496812b9d415b2fa4
I20240502 07:59:39.438131 16531 log_verifier.cc:126] Checking tablet eb6626f239b34d5da46b58f4e849145d
I20240502 07:59:39.535797 16531 log_verifier.cc:177] Verified matching terms for 617 ops in tablet eb6626f239b34d5da46b58f4e849145d
I20240502 07:59:39.535991 16531 log_verifier.cc:126] Checking tablet f4ecc981a63c4488aaa9c58a272b6882
I20240502 07:59:39.651110 16531 log_verifier.cc:177] Verified matching terms for 721 ops in tablet f4ecc981a63c4488aaa9c58a272b6882
I20240502 07:59:39.651347 16531 log_verifier.cc:126] Checking tablet fb030dd86b4a4e50a013a0fee3d8dcd2
I20240502 07:59:39.767838 16531 log_verifier.cc:177] Verified matching terms for 657 ops in tablet fb030dd86b4a4e50a013a0fee3d8dcd2
I20240502 07:59:39.770088 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.200:0
--local_ip_for_outbound_sockets=127.16.36.200
--webserver_interface=127.16.36.200
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--builtin_ntp_servers=127.16.36.212:34767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:39.880939 27080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:39.881163 27080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:39.881215 27080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:39.884330 27080 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:39.884399 27080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:39.884511 27080 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.200
I20240502 07:59:39.888447 27080 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:34767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.200
--webserver_port=0
--tserver_master_addrs=127.16.36.254:45791
--never_fsync=true
--heap_profile_path=/tmp/kudu.27080
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.200
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:39.889393 27080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:39.890569 27080 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:39.899535 27086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:39.900094 27085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:39.900853 27088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:39.901715 27080 server_base.cc:1034] running on GCE node
I20240502 07:59:39.902225 27080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:39.902819 27080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:39.903990 27080 hybrid_clock.cc:648] HybridClock initialized: now 1714636779903972 us; error 28 us; skew 500 ppm
I20240502 07:59:39.906030 27080 webserver.cc:460] Webserver started at http://127.16.36.200:35159/ using document root <none> and password file <none>
I20240502 07:59:39.906598 27080 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:39.906654 27080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:39.906847 27080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:39.908636 27080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "63cdc185b2ad46189853031567ce0a8d"
format_stamp: "Formatted at 2024-05-02 07:59:39 on dist-test-slave-9sjq"
I20240502 07:59:39.909116 27080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "63cdc185b2ad46189853031567ce0a8d"
format_stamp: "Formatted at 2024-05-02 07:59:39 on dist-test-slave-9sjq"
I20240502 07:59:39.912546 27080 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:39.915153 27094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:39.916225 27080 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:39.916348 27080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "63cdc185b2ad46189853031567ce0a8d"
format_stamp: "Formatted at 2024-05-02 07:59:39 on dist-test-slave-9sjq"
I20240502 07:59:39.916445 27080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:39.937407 27080 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:39.938098 27080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:39.938266 27080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:39.938807 27080 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:39.939720 27080 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:39.939769 27080 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:39.939827 27080 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:39.939852 27080 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:39.949045 27080 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.200:41377
I20240502 07:59:39.949071 27207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.200:41377 every 8 connection(s)
I20240502 07:59:39.950030 27080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:59:39.956782 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 27080
I20240502 07:59:39.956948 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 07:59:39.959369 27208 heartbeater.cc:347] Connected to a master server at 127.16.36.254:45791
I20240502 07:59:39.959714 27208 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:39.960287 27208 heartbeater.cc:511] Master 127.16.36.254:45791 requested a full tablet report, sending...
I20240502 07:59:39.961350 22752 ts_manager.cc:194] Registered new tserver with Master: 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377)
I20240502 07:59:39.961979 22752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.200:52759
I20240502 07:59:40.158711 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:40.158789 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.160681 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.162088 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:40.176503 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.177986 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.193298 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.200477 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:40.290483 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:40.292387 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.295670 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.300858 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:40.302906 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.303675 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:40.306150 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.327389 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.369644 24273 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 697, Committed index: 697, Last appended: 2.697, Last appended by leader: 280, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:40.371474 23164 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Refusing update from remote peer 1f7c07da974747dc9d4d1a2c19c54f02: Log matching property violated. Preceding OpId in replica: term: 2 index: 697. Preceding OpId from leader: term: 2 index: 698. (index mismatch)
I20240502 07:59:40.371904 27312 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 698, Last known committed idx: 697, Time since last communication: 0.000s
I20240502 07:59:40.373729 26949 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Committing config change with OpId 2.698: config changed from index 300 to 698, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.375466 23164 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Committing config change with OpId 2.698: config changed from index 300 to 698, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
W20240502 07:59:40.377049 24211 consensus_peers.cc:485] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 -> Peer 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377): Couldn't send request to peer 63cdc185b2ad46189853031567ce0a8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 15fe5361ff8d4e36b6e59b48813f4aaf. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:40.377619 22753 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 reported cstate change: config changed from index 300 to 698, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New cstate: current_term: 2 leader_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" committed_config { opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.394662 23164 consensus_queue.cc:237] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 587, Committed index: 587, Last appended: 1.587, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:40.396385 24538 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 587. Preceding OpId from leader: term: 1 index: 588. (index mismatch)
I20240502 07:59:40.396888 26434 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 588, Last known committed idx: 587, Time since last communication: 0.000s
I20240502 07:59:40.399328 27335 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Committing config change with OpId 1.588: config changed from index 412 to 588, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.399830 24538 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Committing config change with OpId 1.588: config changed from index 412 to 588, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
W20240502 07:59:40.400085 23101 consensus_peers.cc:485] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 -> Peer 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377): Couldn't send request to peer 63cdc185b2ad46189853031567ce0a8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa8e72ea69b147f4925c7cb6cb8ad24d. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:40.402289 22752 catalog_manager.cc:5484] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 reported cstate change: config changed from index 412 to 588, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.421216 24538 consensus_queue.cc:237] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 675, Committed index: 675, Last appended: 2.675, Last appended by leader: 476, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 676 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:40.422817 22900 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 675. Preceding OpId from leader: term: 2 index: 676. (index mismatch)
I20240502 07:59:40.423249 27354 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 676, Last known committed idx: 675, Time since last communication: 0.000s
I20240502 07:59:40.425328 26122 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.676: config changed from index 488 to 676, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 676 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.425848 22900 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Committing config change with OpId 2.676: config changed from index 488 to 676, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 676 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
W20240502 07:59:40.426301 24476 consensus_peers.cc:485] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 -> Peer 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377): Couldn't send request to peer 63cdc185b2ad46189853031567ce0a8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f8ca72508214906a5dcaf459425afe4. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:40.428180 22753 catalog_manager.cc:5484] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 488 to 676, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 676 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.446995 24538 consensus_queue.cc:237] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 617, Committed index: 617, Last appended: 2.617, Last appended by leader: 155, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:40.448359 23032 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 617. Preceding OpId from leader: term: 2 index: 618. (index mismatch)
W20240502 07:59:40.448565 24476 consensus_peers.cc:485] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 -> Peer 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377): Couldn't send request to peer 63cdc185b2ad46189853031567ce0a8d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30773ad41182415ea963b2f59efba756. This is attempt 1: this message will repeat every 5th retry.
I20240502 07:59:40.448833 26122 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 618, Last known committed idx: 617, Time since last communication: 0.000s
I20240502 07:59:40.450459 27354 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.618: config changed from index 529 to 618, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.450857 23032 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Committing config change with OpId 2.618: config changed from index 529 to 618, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New config: { opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.452944 22752 catalog_manager.cc:5484] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 529 to 618, VOTER 63cdc185b2ad46189853031567ce0a8d (127.16.36.200) added. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:40.517678 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.517700 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:40.522536 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.529027 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.529356 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.530922 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:40.542006 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.563732 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:40.640957 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.642243 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.642776 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:40.650327 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:40.650805 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.656678 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.658900 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:40.676621 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.757640 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.759016 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.761212 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.764214 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.764611 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:40.765468 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:40.768580 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:40.788446 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.869683 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:40.870724 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.871012 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.873502 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:40.874183 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.881278 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:40.881279 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:40.903631 27521 ts_tablet_manager.cc:902] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: Initiating tablet copy from peer e53ee470f3824340a097021b74b80353 (127.16.36.199:34991)
I20240502 07:59:40.904491 27521 tablet_copy_client.cc:308] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.199:34991
I20240502 07:59:40.911641 27522 ts_tablet_manager.cc:902] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Initiating tablet copy from peer 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707)
I20240502 07:59:40.912168 27522 tablet_copy_client.cc:308] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.197:34707
I20240502 07:59:40.918417 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:40.921386 24560 tablet_copy_service.cc:140] P e53ee470f3824340a097021b74b80353: Received BeginTabletCopySession request for tablet 30773ad41182415ea963b2f59efba756 from peer 63cdc185b2ad46189853031567ce0a8d ({username='slave'} at 127.16.36.200:50359)
I20240502 07:59:40.921535 24560 tablet_copy_service.cc:161] P e53ee470f3824340a097021b74b80353: Beginning new tablet copy session on tablet 30773ad41182415ea963b2f59efba756 from peer 63cdc185b2ad46189853031567ce0a8d at {username='slave'} at 127.16.36.200:50359: session id = 63cdc185b2ad46189853031567ce0a8d-30773ad41182415ea963b2f59efba756
I20240502 07:59:40.922994 24560 tablet_copy_source_session.cc:205] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:40.926864 27521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30773ad41182415ea963b2f59efba756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.931001 24295 tablet_copy_service.cc:140] P 1f7c07da974747dc9d4d1a2c19c54f02: Received BeginTabletCopySession request for tablet 15fe5361ff8d4e36b6e59b48813f4aaf from peer 63cdc185b2ad46189853031567ce0a8d ({username='slave'} at 127.16.36.200:34647)
I20240502 07:59:40.931151 24295 tablet_copy_service.cc:161] P 1f7c07da974747dc9d4d1a2c19c54f02: Beginning new tablet copy session on tablet 15fe5361ff8d4e36b6e59b48813f4aaf from peer 63cdc185b2ad46189853031567ce0a8d at {username='slave'} at 127.16.36.200:34647: session id = 63cdc185b2ad46189853031567ce0a8d-15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:40.932575 24295 tablet_copy_source_session.cc:205] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:40.933502 27521 tablet_copy_client.cc:783] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:40.933655 27522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15fe5361ff8d4e36b6e59b48813f4aaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.933773 27521 tablet_copy_client.cc:647] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:40.935743 27522 tablet_copy_client.cc:783] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:40.935909 27522 tablet_copy_client.cc:647] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:40.936900 27521 tablet_copy_client.cc:523] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:40.938614 27522 tablet_copy_client.cc:523] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:40.938858 27521 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: Bootstrap starting.
I20240502 07:59:40.939823 27522 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Bootstrap starting.
I20240502 07:59:40.964243 27208 heartbeater.cc:503] Master 127.16.36.254:45791 was elected leader, sending a full tablet report...
I20240502 07:59:40.981760 27544 ts_tablet_manager.cc:902] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: Initiating tablet copy from peer d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213)
I20240502 07:59:40.982462 27544 tablet_copy_client.cc:308] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33213
I20240502 07:59:40.988801 23185 tablet_copy_service.cc:140] P d111283808d34d0d8671028dd2d826d5: Received BeginTabletCopySession request for tablet aa8e72ea69b147f4925c7cb6cb8ad24d from peer 63cdc185b2ad46189853031567ce0a8d ({username='slave'} at 127.16.36.200:41311)
I20240502 07:59:40.988919 23185 tablet_copy_service.cc:161] P d111283808d34d0d8671028dd2d826d5: Beginning new tablet copy session on tablet aa8e72ea69b147f4925c7cb6cb8ad24d from peer 63cdc185b2ad46189853031567ce0a8d at {username='slave'} at 127.16.36.200:41311: session id = 63cdc185b2ad46189853031567ce0a8d-aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:40.989851 23185 tablet_copy_source_session.cc:205] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:40.991518 27544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa8e72ea69b147f4925c7cb6cb8ad24d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:40.993772 27544 tablet_copy_client.cc:783] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:40.993922 27544 tablet_copy_client.cc:647] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:40.997071 27544 tablet_copy_client.cc:523] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:40.997287 27522 log.cc:824] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:40.998457 27544 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: Bootstrap starting.
I20240502 07:59:41.020429 27569 ts_tablet_manager.cc:902] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: Initiating tablet copy from peer e53ee470f3824340a097021b74b80353 (127.16.36.199:34991)
I20240502 07:59:41.025732 27569 tablet_copy_client.cc:308] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.199:34991
I20240502 07:59:41.026281 24560 tablet_copy_service.cc:140] P e53ee470f3824340a097021b74b80353: Received BeginTabletCopySession request for tablet 5f8ca72508214906a5dcaf459425afe4 from peer 63cdc185b2ad46189853031567ce0a8d ({username='slave'} at 127.16.36.200:50359)
I20240502 07:59:41.026392 24560 tablet_copy_service.cc:161] P e53ee470f3824340a097021b74b80353: Beginning new tablet copy session on tablet 5f8ca72508214906a5dcaf459425afe4 from peer 63cdc185b2ad46189853031567ce0a8d at {username='slave'} at 127.16.36.200:50359: session id = 63cdc185b2ad46189853031567ce0a8d-5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:41.027352 24560 tablet_copy_source_session.cc:205] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 07:59:41.029342 27569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8ca72508214906a5dcaf459425afe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:41.031904 27569 tablet_copy_client.cc:783] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 0 data blocks...
I20240502 07:59:41.032142 27569 tablet_copy_client.cc:647] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Starting download of 1 WAL segments...
I20240502 07:59:41.036993 27569 tablet_copy_client.cc:523] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 07:59:41.038506 27569 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: Bootstrap starting.
I20240502 07:59:41.062479 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.086315 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:41.087575 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.087728 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.088841 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.103961 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.104792 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:41.116652 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 07:59:41.134711 27522 log.cc:924] Time spent T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Append to log took a long time: real 0.055s user 0.003s sys 0.000s
I20240502 07:59:41.321192 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:41.328625 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.329056 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.335294 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.362323 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:41.363343 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.386008 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.402666 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:41.570628 27521 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=618 overwritten=0 applied=618 ignored=0} inserts{seen=1246 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:41.571161 27521 tablet_bootstrap.cc:492] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: Bootstrap complete.
I20240502 07:59:41.571693 27521 ts_tablet_manager.cc:1367] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: Time spent bootstrapping tablet: real 0.633s user 0.415s sys 0.028s
I20240502 07:59:41.574569 27521 raft_consensus.cc:348] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.574949 27521 raft_consensus.cc:729] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cdc185b2ad46189853031567ce0a8d, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.575549 27521 consensus_queue.cc:260] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 618, Last appended: 2.618, Last appended by leader: 618, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 618 OBSOLETE_local: false peers { permanent_uuid: "86a82caa98a74858bb3cb7b6b248f65e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38943 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.576646 27521 ts_tablet_manager.cc:1398] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 07:59:41.579603 24560 tablet_copy_service.cc:342] P e53ee470f3824340a097021b74b80353: Request end of tablet copy session 63cdc185b2ad46189853031567ce0a8d-30773ad41182415ea963b2f59efba756 received from {username='slave'} at 127.16.36.200:50359
I20240502 07:59:41.579721 24560 tablet_copy_service.cc:433] P e53ee470f3824340a097021b74b80353: ending tablet copy session 63cdc185b2ad46189853031567ce0a8d-30773ad41182415ea963b2f59efba756 on tablet 30773ad41182415ea963b2f59efba756 with peer 63cdc185b2ad46189853031567ce0a8d
I20240502 07:59:41.625391 27522 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=698 overwritten=0 applied=698 ignored=0} inserts{seen=1326 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:41.625707 27522 tablet_bootstrap.cc:492] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Bootstrap complete.
I20240502 07:59:41.625944 27522 ts_tablet_manager.cc:1367] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Time spent bootstrapping tablet: real 0.686s user 0.465s sys 0.031s
I20240502 07:59:41.626466 27522 raft_consensus.cc:348] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.626577 27522 raft_consensus.cc:729] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cdc185b2ad46189853031567ce0a8d, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.626755 27522 consensus_queue.cc:260] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 698, Last appended: 2.698, Last appended by leader: 698, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.627049 27522 ts_tablet_manager.cc:1398] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:41.627856 24295 tablet_copy_service.cc:342] P 1f7c07da974747dc9d4d1a2c19c54f02: Request end of tablet copy session 63cdc185b2ad46189853031567ce0a8d-15fe5361ff8d4e36b6e59b48813f4aaf received from {username='slave'} at 127.16.36.200:34647
I20240502 07:59:41.627967 24295 tablet_copy_service.cc:433] P 1f7c07da974747dc9d4d1a2c19c54f02: ending tablet copy session 63cdc185b2ad46189853031567ce0a8d-15fe5361ff8d4e36b6e59b48813f4aaf on tablet 15fe5361ff8d4e36b6e59b48813f4aaf with peer 63cdc185b2ad46189853031567ce0a8d
I20240502 07:59:41.668573 27544 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=588 overwritten=0 applied=588 ignored=0} inserts{seen=1258 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:41.669010 27544 tablet_bootstrap.cc:492] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: Bootstrap complete.
I20240502 07:59:41.669329 27544 ts_tablet_manager.cc:1367] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: Time spent bootstrapping tablet: real 0.671s user 0.394s sys 0.033s
I20240502 07:59:41.669858 27544 raft_consensus.cc:348] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.670015 27544 raft_consensus.cc:729] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cdc185b2ad46189853031567ce0a8d, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.670192 27544 consensus_queue.cc:260] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 588, Last appended: 1.588, Last appended by leader: 588, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.670609 27544 ts_tablet_manager.cc:1398] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.671283 23185 tablet_copy_service.cc:342] P d111283808d34d0d8671028dd2d826d5: Request end of tablet copy session 63cdc185b2ad46189853031567ce0a8d-aa8e72ea69b147f4925c7cb6cb8ad24d received from {username='slave'} at 127.16.36.200:41311
I20240502 07:59:41.671365 23185 tablet_copy_service.cc:433] P d111283808d34d0d8671028dd2d826d5: ending tablet copy session 63cdc185b2ad46189853031567ce0a8d-aa8e72ea69b147f4925c7cb6cb8ad24d on tablet aa8e72ea69b147f4925c7cb6cb8ad24d with peer 63cdc185b2ad46189853031567ce0a8d
I20240502 07:59:41.704852 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.705762 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.711000 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.725139 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:41.725198 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.725970 27569 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: Bootstrap replayed 1/1 log segments. Stats: ops{read=676 overwritten=0 applied=676 ignored=0} inserts{seen=1304 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 07:59:41.726361 27569 tablet_bootstrap.cc:492] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: Bootstrap complete.
I20240502 07:59:41.726665 27569 ts_tablet_manager.cc:1367] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: Time spent bootstrapping tablet: real 0.688s user 0.462s sys 0.024s
I20240502 07:59:41.727210 27569 raft_consensus.cc:348] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 676 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.727367 27569 raft_consensus.cc:729] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63cdc185b2ad46189853031567ce0a8d, State: Initialized, Role: FOLLOWER
I20240502 07:59:41.727555 27569 consensus_queue.cc:260] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 676, Last appended: 2.676, Last appended by leader: 676, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 676 OBSOLETE_local: false peers { permanent_uuid: "5309ff82927d47ff824d67ad15631b0a" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46399 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:41.727949 27569 ts_tablet_manager.cc:1398] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:41.728400 24560 tablet_copy_service.cc:342] P e53ee470f3824340a097021b74b80353: Request end of tablet copy session 63cdc185b2ad46189853031567ce0a8d-5f8ca72508214906a5dcaf459425afe4 received from {username='slave'} at 127.16.36.200:50359
I20240502 07:59:41.728513 24560 tablet_copy_service.cc:433] P e53ee470f3824340a097021b74b80353: ending tablet copy session 63cdc185b2ad46189853031567ce0a8d-5f8ca72508214906a5dcaf459425afe4 on tablet 5f8ca72508214906a5dcaf459425afe4 with peer 63cdc185b2ad46189853031567ce0a8d
I20240502 07:59:41.729853 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:41.750962 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:41.753155 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.831977 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.836973 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:41.840385 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:41.844493 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.844602 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.846632 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.861719 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.881805 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:41.913324 27162 raft_consensus.cc:1203] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.675->[2.676-2.676] Dedup: 2.676->[]
I20240502 07:59:41.923262 27162 raft_consensus.cc:1203] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.587->[1.588-1.588] Dedup: 1.588->[]
I20240502 07:59:41.964062 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:41.964916 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.969303 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:41.970832 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.973134 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:41.988687 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:41.989423 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:41.995656 27162 raft_consensus.cc:1203] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.697->[2.698-2.698] Dedup: 2.698->[]
I20240502 07:59:42.003044 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.028630 24273 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf"
dest_uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49664
I20240502 07:59:42.028796 24273 raft_consensus.cc:595] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Received request to transfer leadership
I20240502 07:59:42.083667 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.083701 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:42.095257 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.099874 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.101426 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.102840 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.119104 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:42.120441 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.124189 27162 raft_consensus.cc:1203] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.617->[2.618-2.618] Dedup: 2.618->[]
I20240502 07:59:42.213711 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:42.213809 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.220405 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.220696 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.223490 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.237486 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.242703 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.244978 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:42.276885 23164 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa8e72ea69b147f4925c7cb6cb8ad24d"
dest_uuid: "d111283808d34d0d8671028dd2d826d5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39316
I20240502 07:59:42.277048 23164 raft_consensus.cc:595] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Received request to transfer leadership
I20240502 07:59:42.341444 24255 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.342288 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:42.344306 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.351292 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.358858 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.370049 24520 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:42.370546 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.374732 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.393631 26949 raft_consensus.cc:979] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: : Instructing follower d111283808d34d0d8671028dd2d826d5 to start an election
I20240502 07:59:42.393760 26949 raft_consensus.cc:1067] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Signalling peer d111283808d34d0d8671028dd2d826d5 to start an election
I20240502 07:59:42.394367 23164 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf"
dest_uuid: "d111283808d34d0d8671028dd2d826d5"
from {username='slave'} at 127.16.36.197:58031
I20240502 07:59:42.394551 23164 raft_consensus.cc:482] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:42.394615 23164 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:42.395753 23164 raft_consensus.cc:504] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.396327 23164 leader_election.cc:288] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 3 election: Requested vote from peers 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707), 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377)
I20240502 07:59:42.396766 24273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 698 } } ignore_live_leader: true dest_uuid: "1f7c07da974747dc9d4d1a2c19c54f02"
I20240502 07:59:42.396986 24273 raft_consensus.cc:3032] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 07:59:42.397046 24273 raft_consensus.cc:729] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1f7c07da974747dc9d4d1a2c19c54f02, State: Running, Role: LEADER
I20240502 07:59:42.397228 24273 consensus_queue.cc:260] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 698, Committed index: 698, Last appended: 2.698, Last appended by leader: 698, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.397323 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15fe5361ff8d4e36b6e59b48813f4aaf" candidate_uuid: "d111283808d34d0d8671028dd2d826d5" candidate_term: 3 candidate_status { last_received { term: 2 index: 698 } } ignore_live_leader: true dest_uuid: "63cdc185b2ad46189853031567ce0a8d"
I20240502 07:59:42.397418 24273 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:42.397502 27162 raft_consensus.cc:3037] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Advancing to term 3
I20240502 07:59:42.398547 24273 raft_consensus.cc:2454] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 3.
I20240502 07:59:42.398634 27162 raft_consensus.cc:2454] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d111283808d34d0d8671028dd2d826d5 in term 3.
I20240502 07:59:42.398958 23100 leader_election.cc:302] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f7c07da974747dc9d4d1a2c19c54f02, d111283808d34d0d8671028dd2d826d5; no voters:
I20240502 07:59:42.399298 26434 raft_consensus.cc:2781] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 07:59:42.399452 26434 raft_consensus.cc:686] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 3 LEADER]: Becoming Leader. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:42.399609 26434 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 698, Committed index: 698, Last appended: 2.698, Last appended by leader: 698, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.401616 22752 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 reported cstate change: term changed from 2 to 3, leader changed from 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) to d111283808d34d0d8671028dd2d826d5 (127.16.36.195). New cstate: current_term: 3 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 698 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.413969 27859 raft_consensus.cc:979] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: : Instructing follower e53ee470f3824340a097021b74b80353 to start an election
I20240502 07:59:42.414108 27859 raft_consensus.cc:1067] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Signalling peer e53ee470f3824340a097021b74b80353 to start an election
I20240502 07:59:42.414539 24538 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa8e72ea69b147f4925c7cb6cb8ad24d"
dest_uuid: "e53ee470f3824340a097021b74b80353"
from {username='slave'} at 127.16.36.195:35077
I20240502 07:59:42.414690 24538 raft_consensus.cc:482] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 07:59:42.414743 24538 raft_consensus.cc:3037] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:42.415618 24538 raft_consensus.cc:504] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.415972 24538 leader_election.cc:288] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [CANDIDATE]: Term 2 election: Requested vote from peers d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213), 63cdc185b2ad46189853031567ce0a8d (127.16.36.200:41377)
I20240502 07:59:42.416429 23164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8e72ea69b147f4925c7cb6cb8ad24d" candidate_uuid: "e53ee470f3824340a097021b74b80353" candidate_term: 2 candidate_status { last_received { term: 1 index: 588 } } ignore_live_leader: true dest_uuid: "d111283808d34d0d8671028dd2d826d5"
I20240502 07:59:42.416496 27162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa8e72ea69b147f4925c7cb6cb8ad24d" candidate_uuid: "e53ee470f3824340a097021b74b80353" candidate_term: 2 candidate_status { last_received { term: 1 index: 588 } } ignore_live_leader: true dest_uuid: "63cdc185b2ad46189853031567ce0a8d"
I20240502 07:59:42.416594 23164 raft_consensus.cc:3032] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 07:59:42.416616 27162 raft_consensus.cc:3037] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:42.416642 23164 raft_consensus.cc:729] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d111283808d34d0d8671028dd2d826d5, State: Running, Role: LEADER
I20240502 07:59:42.416800 23164 consensus_queue.cc:260] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 1.588, Last appended by leader: 588, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.416963 23164 raft_consensus.cc:3037] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 07:59:42.417536 27162 raft_consensus.cc:2454] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53ee470f3824340a097021b74b80353 in term 2.
I20240502 07:59:42.417773 23164 raft_consensus.cc:2454] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e53ee470f3824340a097021b74b80353 in term 2.
I20240502 07:59:42.417862 24476 leader_election.cc:302] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63cdc185b2ad46189853031567ce0a8d, e53ee470f3824340a097021b74b80353; no voters:
I20240502 07:59:42.418198 26122 raft_consensus.cc:2781] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 07:59:42.418366 26122 raft_consensus.cc:686] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Becoming Leader. State: Replica: e53ee470f3824340a097021b74b80353, State: Running, Role: LEADER
I20240502 07:59:42.418506 26122 consensus_queue.cc:237] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 588, Committed index: 588, Last appended: 1.588, Last appended by leader: 588, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.420238 22752 catalog_manager.cc:5484] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 reported cstate change: term changed from 1 to 2, leader changed from d111283808d34d0d8671028dd2d826d5 (127.16.36.195) to e53ee470f3824340a097021b74b80353 (127.16.36.199). New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 588 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.678851 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.679662 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.680197 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:42.688843 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.688854 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.694115 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.696815 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:42.704972 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.759207 24538 consensus_queue.cc:237] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 618, Committed index: 618, Last appended: 2.618, Last appended by leader: 155, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 619 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.760619 27162 raft_consensus.cc:1261] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 618. Preceding OpId from leader: term: 2 index: 619. (index mismatch)
I20240502 07:59:42.761087 26122 consensus_queue.cc:1035] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 619, Last known committed idx: 618, Time since last communication: 0.000s
I20240502 07:59:42.763793 27865 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.619: config changed from index 618 to 619, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 619 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.764395 27162 raft_consensus.cc:2932] T 30773ad41182415ea963b2f59efba756 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Committing config change with OpId 2.619: config changed from index 618 to 619, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New config: { opid_index: 619 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.767319 22758 catalog_manager.cc:5484] T 30773ad41182415ea963b2f59efba756 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 618 to 619, VOTER 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 619 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.770099 23014 tablet_service.cc:1514] Processing DeleteTablet for tablet 30773ad41182415ea963b2f59efba756 with delete_type TABLET_DATA_TOMBSTONED (TS 86a82caa98a74858bb3cb7b6b248f65e not found in new config with opid_index 619) from {username='slave'} at 127.0.0.1:50472
I20240502 07:59:42.770506 27923 tablet_replica.cc:330] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: stopping tablet replica
I20240502 07:59:42.770653 27923 raft_consensus.cc:2229] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:42.770802 27923 raft_consensus.cc:2258] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:42.772225 27923 ts_tablet_manager.cc:1875] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:42.774628 27923 ts_tablet_manager.cc:1888] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.618
I20240502 07:59:42.774721 27923 log.cc:1196] T 30773ad41182415ea963b2f59efba756 P 86a82caa98a74858bb3cb7b6b248f65e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/30773ad41182415ea963b2f59efba756
I20240502 07:59:42.775521 22745 catalog_manager.cc:4830] TS 86a82caa98a74858bb3cb7b6b248f65e (127.16.36.194:38943): tablet 30773ad41182415ea963b2f59efba756 (table rebalance_test_table_0 [id=327cde20b38b4a82ba4dc971b0e123be]) successfully deleted
I20240502 07:59:42.839210 23164 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 1 index: 588. Preceding OpId from leader: term: 2 index: 589. (index mismatch)
I20240502 07:59:42.839968 27865 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I20240502 07:59:42.845098 27162 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 1 index: 588. Preceding OpId from leader: term: 2 index: 589. (index mismatch)
I20240502 07:59:42.845564 26122 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 589, Last known committed idx: 588, Time since last communication: 0.000s
I20240502 07:59:42.855794 24273 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 3 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 698. Preceding OpId from leader: term: 3 index: 699. (index mismatch)
I20240502 07:59:42.856086 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:42.856284 27859 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f7c07da974747dc9d4d1a2c19c54f02" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 34707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 699, Last known committed idx: 698, Time since last communication: 0.000s
I20240502 07:59:42.856653 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:42.862454 27162 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 3 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 698. Preceding OpId from leader: term: 3 index: 699. (index mismatch)
I20240502 07:59:42.862802 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.863253 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.863440 26434 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 699, Last known committed idx: 698, Time since last communication: 0.000s
I20240502 07:59:42.871275 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:42.905342 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.913425 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:42.917382 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:42.977535 24538 consensus_queue.cc:237] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 676, Committed index: 676, Last appended: 2.676, Last appended by leader: 476, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:42.978731 27162 raft_consensus.cc:1261] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 676. Preceding OpId from leader: term: 2 index: 677. (index mismatch)
I20240502 07:59:42.979130 26122 consensus_queue.cc:1035] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 677, Last known committed idx: 676, Time since last communication: 0.000s
I20240502 07:59:42.980958 27865 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.677: config changed from index 676 to 677, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.982302 27162 raft_consensus.cc:2932] T 5f8ca72508214906a5dcaf459425afe4 P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Committing config change with OpId 2.677: config changed from index 676 to 677, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New config: { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.983534 22752 catalog_manager.cc:5484] T 5f8ca72508214906a5dcaf459425afe4 P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 676 to 677, VOTER 5309ff82927d47ff824d67ad15631b0a (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 677 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:42.987771 22883 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f8ca72508214906a5dcaf459425afe4 with delete_type TABLET_DATA_TOMBSTONED (TS 5309ff82927d47ff824d67ad15631b0a not found in new config with opid_index 677) from {username='slave'} at 127.0.0.1:39416
I20240502 07:59:42.988294 27991 tablet_replica.cc:330] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: stopping tablet replica
I20240502 07:59:42.988608 27991 raft_consensus.cc:2229] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:42.988891 27991 raft_consensus.cc:2258] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:42.990830 27991 ts_tablet_manager.cc:1875] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:42.996222 27991 ts_tablet_manager.cc:1888] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.676
I20240502 07:59:42.996503 27991 log.cc:1196] T 5f8ca72508214906a5dcaf459425afe4 P 5309ff82927d47ff824d67ad15631b0a: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:42.997450 22747 catalog_manager.cc:4830] TS 5309ff82927d47ff824d67ad15631b0a (127.16.36.193:46399): tablet 5f8ca72508214906a5dcaf459425afe4 (table rebalance_test_table_1 [id=5a5ae1afbfd34a46a0bc4141f7fdfc0c]) successfully deleted
I20240502 07:59:43.053603 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.054309 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:43.063916 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.064591 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.065048 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.065893 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.086803 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.099180 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.143162 23164 consensus_queue.cc:237] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 699, Committed index: 699, Last appended: 3.699, Last appended by leader: 698, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 700 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:43.144342 27162 raft_consensus.cc:1261] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 3 FOLLOWER]: Refusing update from remote peer d111283808d34d0d8671028dd2d826d5: Log matching property violated. Preceding OpId in replica: term: 3 index: 699. Preceding OpId from leader: term: 3 index: 700. (index mismatch)
I20240502 07:59:43.144760 26434 consensus_queue.cc:1035] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 700, Last known committed idx: 699, Time since last communication: 0.000s
I20240502 07:59:43.146396 27859 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P d111283808d34d0d8671028dd2d826d5 [term 3 LEADER]: Committing config change with OpId 3.700: config changed from index 698 to 700, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) evicted. New config: { opid_index: 700 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:43.146862 27162 raft_consensus.cc:2932] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d [term 3 FOLLOWER]: Committing config change with OpId 3.700: config changed from index 698 to 700, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) evicted. New config: { opid_index: 700 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:43.149238 22759 catalog_manager.cc:5484] T 15fe5361ff8d4e36b6e59b48813f4aaf P 63cdc185b2ad46189853031567ce0a8d reported cstate change: config changed from index 698 to 700, VOTER 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197) evicted. New cstate: current_term: 3 leader_uuid: "d111283808d34d0d8671028dd2d826d5" committed_config { opid_index: 700 OBSOLETE_local: false peers { permanent_uuid: "d111283808d34d0d8671028dd2d826d5" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33213 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:43.156504 24255 tablet_service.cc:1514] Processing DeleteTablet for tablet 15fe5361ff8d4e36b6e59b48813f4aaf with delete_type TABLET_DATA_TOMBSTONED (TS 1f7c07da974747dc9d4d1a2c19c54f02 not found in new config with opid_index 700) from {username='slave'} at 127.0.0.1:49742
I20240502 07:59:43.156967 28047 tablet_replica.cc:330] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: stopping tablet replica
I20240502 07:59:43.157302 28047 raft_consensus.cc:2229] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:43.157570 28047 raft_consensus.cc:2258] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:43.159535 28047 ts_tablet_manager.cc:1875] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:43.161854 28047 ts_tablet_manager.cc:1888] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.699
I20240502 07:59:43.161953 28047 log.cc:1196] T 15fe5361ff8d4e36b6e59b48813f4aaf P 1f7c07da974747dc9d4d1a2c19c54f02: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/wals/15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:43.162640 22747 catalog_manager.cc:4830] TS 1f7c07da974747dc9d4d1a2c19c54f02 (127.16.36.197:34707): tablet 15fe5361ff8d4e36b6e59b48813f4aaf (table rebalance_test_table_3 [id=cb2d66610b344739b19afc07df570028]) successfully deleted
I20240502 07:59:43.217945 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.220438 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.223402 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.223727 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.224037 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.230634 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.249336 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.268383 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:43.304965 24538 consensus_queue.cc:237] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 589, Committed index: 589, Last appended: 2.589, Last appended by leader: 588, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } }
I20240502 07:59:43.306083 27162 raft_consensus.cc:1261] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Refusing update from remote peer e53ee470f3824340a097021b74b80353: Log matching property violated. Preceding OpId in replica: term: 2 index: 589. Preceding OpId from leader: term: 2 index: 590. (index mismatch)
I20240502 07:59:43.306468 27865 consensus_queue.cc:1035] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 590, Last known committed idx: 589, Time since last communication: 0.000s
I20240502 07:59:43.307881 26122 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 [term 2 LEADER]: Committing config change with OpId 2.590: config changed from index 588 to 590, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:43.308292 27162 raft_consensus.cc:2932] T aa8e72ea69b147f4925c7cb6cb8ad24d P 63cdc185b2ad46189853031567ce0a8d [term 2 FOLLOWER]: Committing config change with OpId 2.590: config changed from index 588 to 590, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New config: { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:43.310815 22761 catalog_manager.cc:5484] T aa8e72ea69b147f4925c7cb6cb8ad24d P e53ee470f3824340a097021b74b80353 reported cstate change: config changed from index 588 to 590, VOTER d111283808d34d0d8671028dd2d826d5 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "e53ee470f3824340a097021b74b80353" committed_config { opid_index: 590 OBSOLETE_local: false peers { permanent_uuid: "e53ee470f3824340a097021b74b80353" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 34991 } } peers { permanent_uuid: "63cdc185b2ad46189853031567ce0a8d" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 41377 } } }
I20240502 07:59:43.313642 23144 tablet_service.cc:1514] Processing DeleteTablet for tablet aa8e72ea69b147f4925c7cb6cb8ad24d with delete_type TABLET_DATA_TOMBSTONED (TS d111283808d34d0d8671028dd2d826d5 not found in new config with opid_index 590) from {username='slave'} at 127.0.0.1:52558
I20240502 07:59:43.314111 28097 tablet_replica.cc:330] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: stopping tablet replica
I20240502 07:59:43.314251 28097 raft_consensus.cc:2229] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 07:59:43.314386 28097 raft_consensus.cc:2258] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 07:59:43.316043 28097 ts_tablet_manager.cc:1875] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 07:59:43.318367 28097 ts_tablet_manager.cc:1888] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.589
I20240502 07:59:43.318449 28097 log.cc:1196] T aa8e72ea69b147f4925c7cb6cb8ad24d P d111283808d34d0d8671028dd2d826d5: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:43.319137 22748 catalog_manager.cc:4830] TS d111283808d34d0d8671028dd2d826d5 (127.16.36.195:33213): tablet aa8e72ea69b147f4925c7cb6cb8ad24d (table rebalance_test_table_4 [id=8d05709c194a4746bd095e6a09587e29]) successfully deleted
I20240502 07:59:43.596994 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.600713 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.601909 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:43.602950 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.608800 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.608948 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.616839 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.632690 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.717053 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.719094 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:43.719913 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.727648 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.728857 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.729422 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.743177 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.757174 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.848841 24255 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 07:59:43.850346 24387 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 07:59:43.852123 23014 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.855556 27142 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.873099 22883 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 07:59:43.880036 24122 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:43.880354 23144 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:43.884819 24520 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
5229ea5962df43b293bb46e794006e25 | 127.16.36.254:45791 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.212:34767 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
12adf6ad821e4949acc3efc3f6a28404 | 127.16.36.198:38235 | HEALTHY | <none> | 1 | 0
1f7c07da974747dc9d4d1a2c19c54f02 | 127.16.36.197:34707 | HEALTHY | <none> | 2 | 0
2964136ef5e843c18c95afd711c81b09 | 127.16.36.196:42705 | HEALTHY | <none> | 0 | 0
5309ff82927d47ff824d67ad15631b0a | 127.16.36.193:46399 | HEALTHY | <none> | 3 | 0
63cdc185b2ad46189853031567ce0a8d | 127.16.36.200:41377 | HEALTHY | <none> | 0 | 0
86a82caa98a74858bb3cb7b6b248f65e | 127.16.36.194:38943 | HEALTHY | <none> | 3 | 0
d111283808d34d0d8671028dd2d826d5 | 127.16.36.195:33213 | HEALTHY | <none> | 4 | 0
e53ee470f3824340a097021b74b80353 | 127.16.36.199:34991 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.16.36.193 | experimental | 127.16.36.193:46399
local_ip_for_outbound_sockets | 127.16.36.194 | experimental | 127.16.36.194:38943
local_ip_for_outbound_sockets | 127.16.36.195 | experimental | 127.16.36.195:33213
local_ip_for_outbound_sockets | 127.16.36.196 | experimental | 127.16.36.196:42705
local_ip_for_outbound_sockets | 127.16.36.197 | experimental | 127.16.36.197:34707
local_ip_for_outbound_sockets | 127.16.36.198 | experimental | 127.16.36.198:38235
local_ip_for_outbound_sockets | 127.16.36.199 | experimental | 127.16.36.199:34991
local_ip_for_outbound_sockets | 127.16.36.200 | experimental | 127.16.36.200:41377
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.36.193:46399
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.36.194:38943
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.36.195:33213
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.36.196:42705
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.16.36.197:34707
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.16.36.198:38235
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.16.36.199:34991
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalanceParamTest.Rebalance_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.16.36.200:41377
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.212:34767 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 07:59:43.921566 16531 log_verifier.cc:126] Checking tablet 029e2d974d514d6fb63ec3501eb86321
I20240502 07:59:44.021762 16531 log_verifier.cc:177] Verified matching terms for 662 ops in tablet 029e2d974d514d6fb63ec3501eb86321
I20240502 07:59:44.021992 16531 log_verifier.cc:126] Checking tablet 15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:44.133888 16531 log_verifier.cc:177] Verified matching terms for 700 ops in tablet 15fe5361ff8d4e36b6e59b48813f4aaf
I20240502 07:59:44.134130 16531 log_verifier.cc:126] Checking tablet 1c6799e8df114d9c97cf5aafe3233ce2
I20240502 07:59:44.219548 16531 log_verifier.cc:177] Verified matching terms for 543 ops in tablet 1c6799e8df114d9c97cf5aafe3233ce2
I20240502 07:59:44.219731 16531 log_verifier.cc:126] Checking tablet 2bbca5be11d04fed987d979b02773974
I20240502 07:59:44.228932 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 2bbca5be11d04fed987d979b02773974
I20240502 07:59:44.228997 16531 log_verifier.cc:126] Checking tablet 30773ad41182415ea963b2f59efba756
I20240502 07:59:44.336187 16531 log_verifier.cc:177] Verified matching terms for 619 ops in tablet 30773ad41182415ea963b2f59efba756
I20240502 07:59:44.336424 16531 log_verifier.cc:126] Checking tablet 37b107413a8b44c2a72967b33ce4f0ee
I20240502 07:59:44.430541 16531 log_verifier.cc:177] Verified matching terms for 599 ops in tablet 37b107413a8b44c2a72967b33ce4f0ee
I20240502 07:59:44.430739 16531 log_verifier.cc:126] Checking tablet 590e0019653d4f16b8d36cfa6f7ab8b6
I20240502 07:59:44.535543 16531 log_verifier.cc:177] Verified matching terms for 657 ops in tablet 590e0019653d4f16b8d36cfa6f7ab8b6
I20240502 07:59:44.535761 16531 log_verifier.cc:126] Checking tablet 5bf181a9a54e4a05a354ffbcaf020f4c
I20240502 07:59:44.633718 16531 log_verifier.cc:177] Verified matching terms for 664 ops in tablet 5bf181a9a54e4a05a354ffbcaf020f4c
I20240502 07:59:44.633913 16531 log_verifier.cc:126] Checking tablet 5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:44.738374 16531 log_verifier.cc:177] Verified matching terms for 677 ops in tablet 5f8ca72508214906a5dcaf459425afe4
I20240502 07:59:44.738600 16531 log_verifier.cc:126] Checking tablet 6e70ba6d33de4c7eaa320e2dda34ee70
I20240502 07:59:44.839576 16531 log_verifier.cc:177] Verified matching terms for 668 ops in tablet 6e70ba6d33de4c7eaa320e2dda34ee70
I20240502 07:59:44.839793 16531 log_verifier.cc:126] Checking tablet a207b20592e74478beb63c8000b3b1b0
I20240502 07:59:44.946310 16531 log_verifier.cc:177] Verified matching terms for 705 ops in tablet a207b20592e74478beb63c8000b3b1b0
I20240502 07:59:44.946532 16531 log_verifier.cc:126] Checking tablet aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:45.039745 16531 log_verifier.cc:177] Verified matching terms for 590 ops in tablet aa8e72ea69b147f4925c7cb6cb8ad24d
I20240502 07:59:45.039959 16531 log_verifier.cc:126] Checking tablet e91a0e50f7ce40e496812b9d415b2fa4
I20240502 07:59:45.143996 16531 log_verifier.cc:177] Verified matching terms for 667 ops in tablet e91a0e50f7ce40e496812b9d415b2fa4
I20240502 07:59:45.144225 16531 log_verifier.cc:126] Checking tablet eb6626f239b34d5da46b58f4e849145d
I20240502 07:59:45.242225 16531 log_verifier.cc:177] Verified matching terms for 617 ops in tablet eb6626f239b34d5da46b58f4e849145d
I20240502 07:59:45.242484 16531 log_verifier.cc:126] Checking tablet f4ecc981a63c4488aaa9c58a272b6882
I20240502 07:59:45.353893 16531 log_verifier.cc:177] Verified matching terms for 721 ops in tablet f4ecc981a63c4488aaa9c58a272b6882
I20240502 07:59:45.354126 16531 log_verifier.cc:126] Checking tablet fb030dd86b4a4e50a013a0fee3d8dcd2
I20240502 07:59:45.452749 16531 log_verifier.cc:177] Verified matching terms for 657 ops in tablet fb030dd86b4a4e50a013a0fee3d8dcd2
I20240502 07:59:45.454502 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 22821
I20240502 07:59:45.463424 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 22952
I20240502 07:59:45.470600 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 23083
I20240502 07:59:45.477384 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 24059
I20240502 07:59:45.483511 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 24192
I20240502 07:59:45.490056 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 24324
I20240502 07:59:45.496357 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 24457
I20240502 07:59:45.503413 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 27080
I20240502 07:59:45.509455 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 22730
2024-05-02T07:59:45Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/2 (40858 ms)
[----------] 1 test from RebalanceParamTest (40858 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0
I20240502 07:59:45.568195 16531 test_util.cc:268] Using random seed: -58340930
I20240502 07:59:45.569095 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 07:59:45.569151 16531 ts_itest-base.cc:113] --------------
I20240502 07:59:45.569170 16531 ts_itest-base.cc:114] 8 tablet servers
I20240502 07:59:45.569187 16531 ts_itest-base.cc:115] 3 replicas per TS
I20240502 07:59:45.569211 16531 ts_itest-base.cc:116] --------------
2024-05-02T07:59:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T07:59:45Z Disabled control of system clock
I20240502 07:59:45.577821 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:33923
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:33923
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=false
--enable_range_replica_placement=false with env {}
W20240502 07:59:45.690389 28226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:45.690658 28226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:45.690699 28226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:45.694007 28226 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:45.694099 28226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 07:59:45.694125 28226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:45.694144 28226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 07:59:45.694170 28226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 07:59:45.698383 28226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:33923
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:33923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.28226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:45.699415 28226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:45.700721 28226 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 07:59:45.709271 28226 server_base.cc:1034] running on GCE node
W20240502 07:59:45.709964 28232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:45.710629 28234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:45.711364 28231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:45.711966 28226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:45.713088 28226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:45.714259 28226 hybrid_clock.cc:648] HybridClock initialized: now 1714636785714236 us; error 50 us; skew 500 ppm
I20240502 07:59:45.716421 28226 webserver.cc:460] Webserver started at http://127.16.36.254:43601/ using document root <none> and password file <none>
I20240502 07:59:45.717103 28226 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:45.717167 28226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:45.717368 28226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:45.719228 28226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0fa72c150779467baa8c5c75f402ca60"
format_stamp: "Formatted at 2024-05-02 07:59:45 on dist-test-slave-9sjq"
I20240502 07:59:45.719738 28226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0fa72c150779467baa8c5c75f402ca60"
format_stamp: "Formatted at 2024-05-02 07:59:45 on dist-test-slave-9sjq"
I20240502 07:59:45.723250 28226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 07:59:45.725633 28240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:45.726698 28226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:45.726827 28226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0fa72c150779467baa8c5c75f402ca60"
format_stamp: "Formatted at 2024-05-02 07:59:45 on dist-test-slave-9sjq"
I20240502 07:59:45.726933 28226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:45.742208 28226 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:45.742906 28226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:45.743075 28226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:45.750142 28226 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:33923
I20240502 07:59:45.750150 28292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:33923 every 8 connection(s)
I20240502 07:59:45.751164 28226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 07:59:45.753476 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28226
I20240502 07:59:45.753621 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 07:59:45.754016 28293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:45.759876 28293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60: Bootstrap starting.
I20240502 07:59:45.762249 28293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:45.762959 28293 log.cc:824] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:45.764719 28293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60: No bootstrap required, opened a new log
I20240502 07:59:45.767405 28293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa72c150779467baa8c5c75f402ca60" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33923 } }
I20240502 07:59:45.767604 28293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:45.767647 28293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa72c150779467baa8c5c75f402ca60, State: Initialized, Role: FOLLOWER
I20240502 07:59:45.768043 28293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa72c150779467baa8c5c75f402ca60" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33923 } }
I20240502 07:59:45.768164 28293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 07:59:45.768209 28293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 07:59:45.768290 28293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:45.769126 28293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa72c150779467baa8c5c75f402ca60" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33923 } }
I20240502 07:59:45.769443 28293 leader_election.cc:302] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fa72c150779467baa8c5c75f402ca60; no voters:
I20240502 07:59:45.769768 28293 leader_election.cc:288] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 07:59:45.769920 28298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:45.770502 28298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [term 1 LEADER]: Becoming Leader. State: Replica: 0fa72c150779467baa8c5c75f402ca60, State: Running, Role: LEADER
I20240502 07:59:45.770823 28298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa72c150779467baa8c5c75f402ca60" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33923 } }
I20240502 07:59:45.771036 28293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 07:59:45.772248 28299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fa72c150779467baa8c5c75f402ca60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa72c150779467baa8c5c75f402ca60" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33923 } } }
I20240502 07:59:45.772387 28299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [sys.catalog]: This master's current role is: LEADER
I20240502 07:59:45.772735 28300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fa72c150779467baa8c5c75f402ca60. Latest consensus state: current_term: 1 leader_uuid: "0fa72c150779467baa8c5c75f402ca60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa72c150779467baa8c5c75f402ca60" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 33923 } } }
I20240502 07:59:45.772833 28300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60 [sys.catalog]: This master's current role is: LEADER
W20240502 07:59:45.775362 28313 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 07:59:45.775446 28313 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 07:59:45.775532 28309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 07:59:45.776329 28309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 07:59:45.781180 28309 catalog_manager.cc:1347] Generated new cluster ID: fadff55d781740a992aac82fbbb33490
I20240502 07:59:45.781270 28309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 07:59:45.789135 28309 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 07:59:45.790450 28309 catalog_manager.cc:1504] Loading token signing keys...
I20240502 07:59:45.804376 28309 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 0fa72c150779467baa8c5c75f402ca60: Generated new TSK 0
I20240502 07:59:45.805316 28309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 07:59:45.809101 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:45.918783 28317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:45.919034 28317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:45.919113 28317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:45.922323 28317 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:45.922415 28317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:45.922508 28317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 07:59:45.926370 28317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.28317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:45.927314 28317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:45.928577 28317 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:45.938347 28322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:45.938827 28323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:45.939540 28325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:45.940548 28317 server_base.cc:1034] running on GCE node
I20240502 07:59:45.941004 28317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:45.941625 28317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:45.942811 28317 hybrid_clock.cc:648] HybridClock initialized: now 1714636785942796 us; error 32 us; skew 500 ppm
I20240502 07:59:45.944977 28317 webserver.cc:460] Webserver started at http://127.16.36.193:45231/ using document root <none> and password file <none>
I20240502 07:59:45.945580 28317 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:45.945644 28317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:45.945879 28317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:45.947597 28317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "71922d97e1a4414698ecd14a7b2dedec"
format_stamp: "Formatted at 2024-05-02 07:59:45 on dist-test-slave-9sjq"
I20240502 07:59:45.948093 28317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "71922d97e1a4414698ecd14a7b2dedec"
format_stamp: "Formatted at 2024-05-02 07:59:45 on dist-test-slave-9sjq"
I20240502 07:59:45.951606 28317 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 07:59:45.954062 28331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:45.955159 28317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:45.955277 28317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "71922d97e1a4414698ecd14a7b2dedec"
format_stamp: "Formatted at 2024-05-02 07:59:45 on dist-test-slave-9sjq"
I20240502 07:59:45.955375 28317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:45.972388 28317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:45.973174 28317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:45.973361 28317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:45.973928 28317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:45.974850 28317 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:45.974903 28317 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:45.974965 28317 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:45.974990 28317 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:45.983878 28317 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:33537
I20240502 07:59:45.983894 28444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:33537 every 8 connection(s)
I20240502 07:59:45.984968 28317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 07:59:45.985216 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28317
I20240502 07:59:45.985337 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 07:59:45.987823 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:45.995267 28445 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:45.995545 28445 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:45.996064 28445 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:45.997812 28257 ts_manager.cc:194] Registered new tserver with Master: 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:45.999622 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:43669
W20240502 07:59:46.102630 28448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:46.102891 28448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:46.102937 28448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:46.106187 28448 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:46.106268 28448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:46.106340 28448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 07:59:46.110167 28448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.28448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:46.111130 28448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:46.112246 28448 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:46.121448 28456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.122153 28453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.122831 28454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:46.123617 28448 server_base.cc:1034] running on GCE node
I20240502 07:59:46.124048 28448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:46.124735 28448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:46.125917 28448 hybrid_clock.cc:648] HybridClock initialized: now 1714636786125904 us; error 30 us; skew 500 ppm
I20240502 07:59:46.128042 28448 webserver.cc:460] Webserver started at http://127.16.36.194:33321/ using document root <none> and password file <none>
I20240502 07:59:46.128686 28448 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:46.128757 28448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:46.128973 28448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:46.130776 28448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.131304 28448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.134867 28448 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:46.137302 28462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.138394 28448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:46.138520 28448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.138623 28448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:46.162698 28448 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:46.163486 28448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:46.163671 28448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:46.164260 28448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:46.165258 28448 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:46.165313 28448 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.165375 28448 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:46.165408 28448 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.174461 28448 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:40581
I20240502 07:59:46.174513 28575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:40581 every 8 connection(s)
I20240502 07:59:46.175467 28448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 07:59:46.184756 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28448
I20240502 07:59:46.184895 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 07:59:46.185073 28576 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:46.185333 28576 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:46.185917 28576 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:46.186967 28257 ts_manager.cc:194] Registered new tserver with Master: c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:46.187259 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:46.187587 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:37233
W20240502 07:59:46.299002 28579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:46.299268 28579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:46.299319 28579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:46.302623 28579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:46.302719 28579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:46.302798 28579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 07:59:46.306985 28579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.28579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:46.307960 28579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:46.309170 28579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:46.318605 28585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.319128 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.319686 28587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:46.320711 28579 server_base.cc:1034] running on GCE node
I20240502 07:59:46.321195 28579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:46.321849 28579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:46.323009 28579 hybrid_clock.cc:648] HybridClock initialized: now 1714636786322997 us; error 31 us; skew 500 ppm
I20240502 07:59:46.335186 28579 webserver.cc:460] Webserver started at http://127.16.36.195:45107/ using document root <none> and password file <none>
I20240502 07:59:46.335812 28579 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:46.335884 28579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:46.336069 28579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:46.337919 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.338443 28579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.342033 28579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:46.344615 28593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.345772 28579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:46.345894 28579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.346009 28579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:46.361069 28579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:46.361718 28579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:46.361888 28579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:46.362437 28579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:46.363327 28579 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:46.363384 28579 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.363443 28579 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:46.363472 28579 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.372299 28579 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:33991
I20240502 07:59:46.372357 28706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:33991 every 8 connection(s)
I20240502 07:59:46.373358 28579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 07:59:46.374346 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28579
I20240502 07:59:46.374480 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 07:59:46.376634 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:46.383152 28707 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:46.383378 28707 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:46.383828 28707 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:46.384891 28257 ts_manager.cc:194] Registered new tserver with Master: ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:46.385743 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:58707
W20240502 07:59:46.488659 28710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:46.488916 28710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:46.488963 28710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:46.492017 28710 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:46.492090 28710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:46.492161 28710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 07:59:46.495895 28710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.28710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:46.496876 28710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:46.498014 28710 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:46.507123 28716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.507681 28718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.508605 28715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:46.509480 28710 server_base.cc:1034] running on GCE node
I20240502 07:59:46.510046 28710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:46.510690 28710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:46.511899 28710 hybrid_clock.cc:648] HybridClock initialized: now 1714636786511853 us; error 43 us; skew 500 ppm
I20240502 07:59:46.514025 28710 webserver.cc:460] Webserver started at http://127.16.36.196:35459/ using document root <none> and password file <none>
I20240502 07:59:46.514598 28710 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:46.514663 28710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:46.514842 28710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:46.516590 28710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "de423605477a4819a9569821f59f03e4"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.517099 28710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "de423605477a4819a9569821f59f03e4"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.520714 28710 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:46.523048 28724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.524098 28710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:46.524219 28710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "de423605477a4819a9569821f59f03e4"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.524312 28710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:46.550884 28710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:46.551513 28710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:46.551680 28710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:46.552218 28710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:46.553212 28710 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:46.553265 28710 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.553328 28710 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:46.553354 28710 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.561960 28710 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:33237
I20240502 07:59:46.561985 28837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:33237 every 8 connection(s)
I20240502 07:59:46.562965 28710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 07:59:46.563191 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28710
I20240502 07:59:46.563328 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 07:59:46.565821 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:0
--local_ip_for_outbound_sockets=127.16.36.197
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:46.573638 28838 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:46.573908 28838 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:46.574390 28838 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:46.575426 28257 ts_manager.cc:194] Registered new tserver with Master: de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:46.576239 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:39813
W20240502 07:59:46.678617 28841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:46.678880 28841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:46.678925 28841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:46.682089 28841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:46.682166 28841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:46.682236 28841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:59:46.686080 28841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.28841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:46.687031 28841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:46.688144 28841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:46.697049 28846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.697974 28849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.698706 28847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:46.699429 28841 server_base.cc:1034] running on GCE node
I20240502 07:59:46.699864 28841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:46.700562 28841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:46.701735 28841 hybrid_clock.cc:648] HybridClock initialized: now 1714636786701721 us; error 40 us; skew 500 ppm
I20240502 07:59:46.703824 28841 webserver.cc:460] Webserver started at http://127.16.36.197:43737/ using document root <none> and password file <none>
I20240502 07:59:46.704394 28841 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:46.704480 28841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:46.704682 28841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:46.706353 28841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6c68c916f39a45f697df7eaf8973225a"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.706879 28841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6c68c916f39a45f697df7eaf8973225a"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.710526 28841 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 07:59:46.712940 28855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.714090 28841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:46.714200 28841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6c68c916f39a45f697df7eaf8973225a"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.714304 28841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:46.740979 28841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:46.741614 28841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:46.741781 28841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:46.742350 28841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:46.743269 28841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:46.743326 28841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.743384 28841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:46.743420 28841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.752786 28841 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:35407
I20240502 07:59:46.752828 28968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:35407 every 8 connection(s)
I20240502 07:59:46.753926 28841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:46.762913 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28841
I20240502 07:59:46.763067 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 07:59:46.763487 28969 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:46.763708 28969 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:46.764168 28969 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:46.765213 28257 ts_manager.cc:194] Registered new tserver with Master: 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 07:59:46.765332 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.198:0
--local_ip_for_outbound_sockets=127.16.36.198
--webserver_interface=127.16.36.198
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:46.765980 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:45029
W20240502 07:59:46.879232 28972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:46.879556 28972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:46.879647 28972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:46.883136 28972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:46.883246 28972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:46.883349 28972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.198
I20240502 07:59:46.887208 28972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.198
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.28972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.198
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:46.888173 28972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:46.889371 28972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:46.898370 28980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.898851 28978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:46.899506 28977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:46.900543 28972 server_base.cc:1034] running on GCE node
I20240502 07:59:46.901000 28972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:46.901602 28972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:46.902762 28972 hybrid_clock.cc:648] HybridClock initialized: now 1714636786902750 us; error 28 us; skew 500 ppm
I20240502 07:59:46.904837 28972 webserver.cc:460] Webserver started at http://127.16.36.198:35347/ using document root <none> and password file <none>
I20240502 07:59:46.905400 28972 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:46.905455 28972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:46.905624 28972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:46.907316 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6efe17dcca7b4babad4785d19552b699"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.907799 28972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6efe17dcca7b4babad4785d19552b699"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.911190 28972 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 07:59:46.913434 28986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.914431 28972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:46.914552 28972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6efe17dcca7b4babad4785d19552b699"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:46.914641 28972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:46.928344 28972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:46.928983 28972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:46.929155 28972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:46.929697 28972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:46.930583 28972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:46.930632 28972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.930693 28972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:46.930724 28972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:46.939368 28972 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.198:45421
I20240502 07:59:46.939406 29099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.198:45421 every 8 connection(s)
I20240502 07:59:46.940358 28972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:46.941815 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 28972
I20240502 07:59:46.941954 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 07:59:46.944424 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.199:0
--local_ip_for_outbound_sockets=127.16.36.199
--webserver_interface=127.16.36.199
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:46.951010 29100 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:46.951331 29100 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:46.952033 29100 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:46.953361 28257 ts_manager.cc:194] Registered new tserver with Master: 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 07:59:46.953972 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.198:58817
I20240502 07:59:47.002602 28445 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
W20240502 07:59:47.059028 29103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:47.059316 29103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:47.059397 29103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:47.062947 29103 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:47.063052 29103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:47.063159 29103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.199
I20240502 07:59:47.066987 29103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.199
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.29103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.199
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:47.067968 29103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:47.069115 29103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:47.077827 29111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:47.078439 29108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:47.079156 29109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:47.079921 29103 server_base.cc:1034] running on GCE node
I20240502 07:59:47.080358 29103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:47.081017 29103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:47.082226 29103 hybrid_clock.cc:648] HybridClock initialized: now 1714636787082201 us; error 40 us; skew 500 ppm
I20240502 07:59:47.084278 29103 webserver.cc:460] Webserver started at http://127.16.36.199:36455/ using document root <none> and password file <none>
I20240502 07:59:47.084883 29103 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:47.084949 29103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:47.085125 29103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:47.086802 29103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "29126d5aee924f81bdd47cc7153420a1"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:47.087291 29103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "29126d5aee924f81bdd47cc7153420a1"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:47.090682 29103 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:47.093065 29117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:47.094053 29103 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 07:59:47.094174 29103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "29126d5aee924f81bdd47cc7153420a1"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:47.094269 29103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:47.122975 29103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:47.123581 29103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:47.123742 29103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:47.124281 29103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:47.125216 29103 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:47.125268 29103 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:47.125329 29103 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:47.125355 29103 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:47.134505 29103 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.199:43589
I20240502 07:59:47.134526 29230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.199:43589 every 8 connection(s)
I20240502 07:59:47.135480 29103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:59:47.141371 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 29103
I20240502 07:59:47.141516 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 07:59:47.144075 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.200:0
--local_ip_for_outbound_sockets=127.16.36.200
--webserver_interface=127.16.36.200
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:47.145215 29231 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:47.145470 29231 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:47.145992 29231 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:47.146991 28257 ts_manager.cc:194] Registered new tserver with Master: 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589)
I20240502 07:59:47.147545 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.199:56905
I20240502 07:59:47.190348 28576 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
W20240502 07:59:47.261824 29234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:47.262109 29234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:47.262183 29234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:47.265595 29234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:47.265702 29234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:47.265805 29234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.200
I20240502 07:59:47.269788 29234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.200
--webserver_port=0
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.29234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.200
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:47.270825 29234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:47.272254 29234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:47.282167 29239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:47.282590 29242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:47.283221 29240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:47.284255 29234 server_base.cc:1034] running on GCE node
I20240502 07:59:47.284739 29234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:47.285337 29234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:47.286518 29234 hybrid_clock.cc:648] HybridClock initialized: now 1714636787286502 us; error 34 us; skew 500 ppm
I20240502 07:59:47.288663 29234 webserver.cc:460] Webserver started at http://127.16.36.200:33657/ using document root <none> and password file <none>
I20240502 07:59:47.289239 29234 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:47.289299 29234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:47.289505 29234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 07:59:47.291229 29234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "e5e9553dabc94f218f6427c1685ffc38"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:47.291712 29234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "e5e9553dabc94f218f6427c1685ffc38"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:47.295274 29234 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 07:59:47.297744 29248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:47.298957 29234 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:47.299077 29234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e5e9553dabc94f218f6427c1685ffc38"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:47.299181 29234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:47.323293 29234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:47.323913 29234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:47.324083 29234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:47.324662 29234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:47.325559 29234 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:47.325606 29234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:47.325652 29234 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:47.325685 29234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:47.334393 29234 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.200:43189
I20240502 07:59:47.334410 29361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.200:43189 every 8 connection(s)
I20240502 07:59:47.336892 29234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:59:47.341545 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 29234
I20240502 07:59:47.341686 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 07:59:47.347348 29362 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:47.347586 29362 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:47.348052 29362 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:47.349190 28257 ts_manager.cc:194] Registered new tserver with Master: e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
I20240502 07:59:47.349845 28257 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.200:55991
I20240502 07:59:47.357201 16531 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 07:59:47.367658 16531 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 07:59:47.367771 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28841
I20240502 07:59:47.373579 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28972
I20240502 07:59:47.381281 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 29103
I20240502 07:59:47.386492 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 29234
I20240502 07:59:47.387758 28707 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
I20240502 07:59:47.578408 28838 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
I20240502 07:59:51.144239 28257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:51.166764 28771 tablet_service.cc:1467] Processing CreateTablet for tablet ab785e07fa8248e5889bea8ac755d002 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.170042 28378 tablet_service.cc:1467] Processing CreateTablet for tablet c65fe7f823df4b4996769fc784d3adb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.170042 28641 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.170578 28509 tablet_service.cc:1467] Processing CreateTablet for tablet ab785e07fa8248e5889bea8ac755d002 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.171223 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65fe7f823df4b4996769fc784d3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.171223 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba62cb7ab7b429d85e0ac037c127cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.171608 28377 tablet_service.cc:1467] Processing CreateTablet for tablet ab785e07fa8248e5889bea8ac755d002 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.171890 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab785e07fa8248e5889bea8ac755d002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.171845 28640 tablet_service.cc:1467] Processing CreateTablet for tablet c65fe7f823df4b4996769fc784d3adb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.172132 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65fe7f823df4b4996769fc784d3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.172688 28510 tablet_service.cc:1467] Processing CreateTablet for tablet c65fe7f823df4b4996769fc784d3adb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.172996 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.173636 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba62cb7ab7b429d85e0ac037c127cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.173688 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab785e07fa8248e5889bea8ac755d002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.175024 28379 tablet_service.cc:1467] Processing CreateTablet for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.175288 28379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba62cb7ab7b429d85e0ac037c127cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.180909 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65fe7f823df4b4996769fc784d3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.181150 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab785e07fa8248e5889bea8ac755d002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.181293 29385 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:51.184036 29383 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:51.184598 29384 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:51.185261 29385 tablet_bootstrap.cc:654] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.185982 29382 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:51.186200 29385 log.cc:824] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:51.187709 29383 tablet_bootstrap.cc:654] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.187714 29384 tablet_bootstrap.cc:654] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.188349 29382 tablet_bootstrap.cc:654] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.188485 29383 log.cc:824] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:51.188520 29384 log.cc:824] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:51.189152 29382 log.cc:824] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Log is configured to *not* fsync() on all Append() calls
I20240502 07:59:51.190398 29385 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:51.190398 29383 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:51.190409 29384 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:51.190591 29384 ts_tablet_manager.cc:1367] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 07:59:51.190591 29383 ts_tablet_manager.cc:1367] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 07:59:51.190661 29382 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:51.190639 29385 ts_tablet_manager.cc:1367] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 07:59:51.190822 29382 ts_tablet_manager.cc:1367] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 07:59:51.194777 29385 raft_consensus.cc:348] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.194774 29383 raft_consensus.cc:348] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.194774 29384 raft_consensus.cc:348] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.194777 29382 raft_consensus.cc:348] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.195040 29382 raft_consensus.cc:374] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.195040 29384 raft_consensus.cc:374] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.195041 29383 raft_consensus.cc:374] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.195103 29383 raft_consensus.cc:729] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.195113 29384 raft_consensus.cc:729] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.195112 29382 raft_consensus.cc:729] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.195369 29385 raft_consensus.cc:374] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.195475 29385 raft_consensus.cc:729] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.195600 29384 consensus_queue.cc:260] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.195600 29382 consensus_queue.cc:260] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.195603 29383 consensus_queue.cc:260] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.196048 29385 consensus_queue.cc:260] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.196609 29384 ts_tablet_manager.cc:1398] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 07:59:51.196648 29383 ts_tablet_manager.cc:1398] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20240502 07:59:51.196859 29382 ts_tablet_manager.cc:1398] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20240502 07:59:51.197898 29382 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:51.197976 29383 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:51.198279 29384 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:51.199131 29383 tablet_bootstrap.cc:654] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.199210 29384 tablet_bootstrap.cc:654] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.201088 29382 tablet_bootstrap.cc:654] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.201229 29385 ts_tablet_manager.cc:1398] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 07:59:51.201372 29384 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:51.201464 29384 ts_tablet_manager.cc:1367] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:51.201517 29385 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:51.202428 29383 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:51.202514 29383 ts_tablet_manager.cc:1367] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:59:51.202827 29384 raft_consensus.cc:348] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.202939 29384 raft_consensus.cc:374] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.202971 29384 raft_consensus.cc:729] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.203128 29384 consensus_queue.cc:260] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.203099 29383 raft_consensus.cc:348] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.203210 29385 tablet_bootstrap.cc:654] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.203233 29383 raft_consensus.cc:374] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.203280 29383 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.203401 29383 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.203516 29384 ts_tablet_manager.cc:1398] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:51.203673 29383 ts_tablet_manager.cc:1398] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:51.205221 29385 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:51.205332 29385 ts_tablet_manager.cc:1367] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:51.205410 29382 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:51.205500 29382 ts_tablet_manager.cc:1367] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:59:51.205931 29385 raft_consensus.cc:348] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.206049 29385 raft_consensus.cc:374] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.206082 29385 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.206100 29382 raft_consensus.cc:348] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.206220 29382 raft_consensus.cc:374] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.206248 29382 raft_consensus.cc:729] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.206192 29385 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.206367 29382 consensus_queue.cc:260] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.206568 29385 ts_tablet_manager.cc:1398] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.206617 29382 ts_tablet_manager.cc:1398] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.206744 29382 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:51.207655 29382 tablet_bootstrap.cc:654] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.208848 29382 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:51.208926 29382 ts_tablet_manager.cc:1367] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:51.209308 29382 raft_consensus.cc:348] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.209399 29382 raft_consensus.cc:374] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.209420 29382 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.209529 29382 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.209791 29382 ts_tablet_manager.cc:1398] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 07:59:51.240268 28446 tablet.cc:2369] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 07:59:51.318046 28839 tablet.cc:2369] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:51.339319 29390 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:51.339531 29390 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.340772 29390 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:51.345039 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:51.345036 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:51.345331 28530 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:51.345332 28661 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:51.345803 28334 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:51.346066 29390 raft_consensus.cc:2781] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:51.346155 29390 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:51.346205 29390 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.347110 29390 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.347618 29390 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:51.347944 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:51.348006 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:51.348095 28530 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.348114 28661 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.349292 28661 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:51.349293 28530 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:51.349622 28332 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:51.349967 29390 raft_consensus.cc:2781] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:51.350183 29390 raft_consensus.cc:686] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 07:59:51.350523 29390 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.353689 28257 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
W20240502 07:59:51.378003 28708 tablet.cc:2369] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:51.385839 29393 raft_consensus.cc:482] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:51.386006 29393 raft_consensus.cc:504] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.387135 29393 leader_election.cc:288] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:51.391187 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:51.391175 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:51.391438 28399 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:51.391487 28792 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:51.391919 28464 leader_election.cc:302] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:51.392166 29393 raft_consensus.cc:2781] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:51.392235 29393 raft_consensus.cc:482] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:51.392302 29393 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.393400 29393 raft_consensus.cc:504] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.393769 29393 leader_election.cc:288] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:51.394170 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:51.394181 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:51.394317 28399 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.394318 28792 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.395155 28792 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:51.395155 28399 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:51.395466 28464 leader_election.cc:302] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:51.395640 29393 raft_consensus.cc:2781] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:51.395869 29393 raft_consensus.cc:686] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:51.396171 29393 consensus_queue.cc:237] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.398808 28257 catalog_manager.cc:5484] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 07:59:51.431030 28577 tablet.cc:2369] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 07:59:51.431263 29392 raft_consensus.cc:482] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:51.431375 29392 raft_consensus.cc:504] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.432543 29392 leader_election.cc:288] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:51.436167 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba62cb7ab7b429d85e0ac037c127cb4" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:51.436321 28792 raft_consensus.cc:2454] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:51.436296 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba62cb7ab7b429d85e0ac037c127cb4" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:51.436472 28399 raft_consensus.cc:2454] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:51.437047 28597 leader_election.cc:302] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:51.437417 29392 raft_consensus.cc:2781] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:51.437484 29392 raft_consensus.cc:482] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:51.437515 29392 raft_consensus.cc:3037] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.438442 29392 raft_consensus.cc:504] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.438822 29392 leader_election.cc:288] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:51.439208 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba62cb7ab7b429d85e0ac037c127cb4" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:51.439319 28399 raft_consensus.cc:3037] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.439266 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ba62cb7ab7b429d85e0ac037c127cb4" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:51.439411 28792 raft_consensus.cc:3037] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.440150 28399 raft_consensus.cc:2454] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:51.440250 28792 raft_consensus.cc:2454] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:51.440559 28595 leader_election.cc:302] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:51.440747 29392 raft_consensus.cc:2781] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:51.440981 29392 raft_consensus.cc:686] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:51.441301 29392 consensus_queue.cc:237] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:51.444655 28257 catalog_manager.cc:5484] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
W20240502 07:59:51.593262 29411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:51.593567 29411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:51.596783 29411 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 07:59:51.702739 28399 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:51.702740 28792 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:51.703670 29393 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:51.704173 29393 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:51.704492 28661 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:51.704665 28530 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:51.705108 29390 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:51.705529 29401 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:51.713975 29440 mvcc.cc:205] Tried to move back new op lower bound from 7023152298813394944 to 7023152297374654464. Current Snapshot: MvccSnapshot[applied={T|T < 7023152298813394944}]
I20240502 07:59:51.715873 29439 mvcc.cc:205] Tried to move back new op lower bound from 7023152298805477376 to 7023152297561300992. Current Snapshot: MvccSnapshot[applied={T|T < 7023152298805477376}]
I20240502 07:59:51.732978 29449 mvcc.cc:205] Tried to move back new op lower bound from 7023152298813394944 to 7023152297374654464. Current Snapshot: MvccSnapshot[applied={T|T < 7023152298813394944}]
I20240502 07:59:51.737666 28399 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:51.737733 28792 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:51.738209 29409 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:51.738418 29392 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 139.013 ms
time per row: 0.0695063 ms
I20240502 07:59:51.785920 28257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:51.791913 28641 tablet_service.cc:1467] Processing CreateTablet for tablet a7fd7f8583e14c91a932767ebe23b419 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.792071 28509 tablet_service.cc:1467] Processing CreateTablet for tablet a7fd7f8583e14c91a932767ebe23b419 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.792305 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fd7f8583e14c91a932767ebe23b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.792371 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fd7f8583e14c91a932767ebe23b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.792541 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 14bbfb3758e04aeab0df4bbab8380061 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.792778 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bbfb3758e04aeab0df4bbab8380061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.793040 28639 tablet_service.cc:1467] Processing CreateTablet for tablet d08ad2a981fc4c13bc89d22feea7d556 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.793264 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ad2a981fc4c13bc89d22feea7d556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.794978 28771 tablet_service.cc:1467] Processing CreateTablet for tablet a7fd7f8583e14c91a932767ebe23b419 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.795253 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fd7f8583e14c91a932767ebe23b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.795248 29467 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:51.795289 29468 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:51.795673 28378 tablet_service.cc:1467] Processing CreateTablet for tablet d08ad2a981fc4c13bc89d22feea7d556 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.795920 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ad2a981fc4c13bc89d22feea7d556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.796175 29468 tablet_bootstrap.cc:654] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.796175 29467 tablet_bootstrap.cc:654] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.797741 28510 tablet_service.cc:1467] Processing CreateTablet for tablet d08ad2a981fc4c13bc89d22feea7d556 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.797861 29467 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:51.797937 29467 ts_tablet_manager.cc:1367] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:51.798005 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ad2a981fc4c13bc89d22feea7d556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.798421 29467 raft_consensus.cc:348] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.798530 29467 raft_consensus.cc:374] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.798559 29467 raft_consensus.cc:729] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.798677 29467 consensus_queue.cc:260] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.798744 29472 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:51.798938 29467 ts_tablet_manager.cc:1398] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.799393 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 14bbfb3758e04aeab0df4bbab8380061 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.799690 29472 tablet_bootstrap.cc:654] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.798754 28771 tablet_service.cc:1467] Processing CreateTablet for tablet 14bbfb3758e04aeab0df4bbab8380061 (DEFAULT_TABLE table=rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:51.800386 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bbfb3758e04aeab0df4bbab8380061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.800930 29470 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:51.800936 29472 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:51.801005 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bbfb3758e04aeab0df4bbab8380061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:51.801021 29472 ts_tablet_manager.cc:1367] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:51.801466 29472 raft_consensus.cc:348] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.801569 29472 raft_consensus.cc:374] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.801599 29472 raft_consensus.cc:729] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.801749 29472 consensus_queue.cc:260] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.802026 29470 tablet_bootstrap.cc:654] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.802122 29472 ts_tablet_manager.cc:1398] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.802753 29467 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:51.803659 29468 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:51.803761 29468 ts_tablet_manager.cc:1367] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 07:59:51.803949 29467 tablet_bootstrap.cc:654] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.804291 29468 raft_consensus.cc:348] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.804411 29468 raft_consensus.cc:374] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.804472 29468 raft_consensus.cc:729] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.804631 29468 consensus_queue.cc:260] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.804930 29468 ts_tablet_manager.cc:1398] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.805094 29468 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:51.805362 29470 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:51.805447 29470 ts_tablet_manager.cc:1367] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:51.805935 29470 raft_consensus.cc:348] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.806027 29470 raft_consensus.cc:374] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.806062 29468 tablet_bootstrap.cc:654] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.806063 29470 raft_consensus.cc:729] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.806193 29470 consensus_queue.cc:260] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.807013 29470 ts_tablet_manager.cc:1398] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:51.807173 29470 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:51.807336 29467 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:51.807439 29467 ts_tablet_manager.cc:1367] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 07:59:51.807960 29467 raft_consensus.cc:348] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.808079 29467 raft_consensus.cc:374] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.808117 29467 raft_consensus.cc:729] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.808136 29470 tablet_bootstrap.cc:654] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.808244 29467 consensus_queue.cc:260] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.808583 29467 ts_tablet_manager.cc:1398] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:51.808768 29467 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:51.809350 29468 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:51.809444 29468 ts_tablet_manager.cc:1367] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:51.809725 29467 tablet_bootstrap.cc:654] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.809856 29468 raft_consensus.cc:348] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.809950 29468 raft_consensus.cc:374] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.809989 29468 raft_consensus.cc:729] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.810132 29468 consensus_queue.cc:260] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.810644 29468 ts_tablet_manager.cc:1398] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.810822 29468 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:51.811194 29467 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:51.811269 29467 ts_tablet_manager.cc:1367] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:51.811790 29468 tablet_bootstrap.cc:654] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:51.811764 29467 raft_consensus.cc:348] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.812045 29467 raft_consensus.cc:374] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.812093 29467 raft_consensus.cc:729] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.812227 29467 consensus_queue.cc:260] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.812951 29468 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:51.813050 29468 ts_tablet_manager.cc:1367] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:51.813531 29470 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:51.813491 29468 raft_consensus.cc:348] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.813585 29468 raft_consensus.cc:374] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.813624 29468 raft_consensus.cc:729] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.813628 29470 ts_tablet_manager.cc:1367] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:59:51.813771 29467 ts_tablet_manager.cc:1398] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:51.813737 29468 consensus_queue.cc:260] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.813998 29468 ts_tablet_manager.cc:1398] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:51.814060 29470 raft_consensus.cc:348] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.814177 29470 raft_consensus.cc:374] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:51.814225 29470 raft_consensus.cc:729] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:51.814365 29470 consensus_queue.cc:260] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.814630 29470 ts_tablet_manager.cc:1398] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:51.924494 29405 raft_consensus.cc:482] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:51.924634 29405 raft_consensus.cc:504] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.925118 29405 leader_election.cc:288] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:51.925596 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fd7f8583e14c91a932767ebe23b419" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:51.925829 28792 raft_consensus.cc:2454] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:51.926309 28466 leader_election.cc:302] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:51.926501 29405 raft_consensus.cc:2781] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:51.926570 29405 raft_consensus.cc:482] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:51.926625 29405 raft_consensus.cc:3037] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.927636 29405 raft_consensus.cc:504] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.928068 29405 leader_election.cc:288] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:51.928524 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fd7f8583e14c91a932767ebe23b419" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:51.928663 28792 raft_consensus.cc:3037] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.929270 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fd7f8583e14c91a932767ebe23b419" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:51.929318 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7fd7f8583e14c91a932767ebe23b419" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:51.929458 28661 raft_consensus.cc:2454] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:51.929926 28792 raft_consensus.cc:2454] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:51.930334 28466 leader_election.cc:302] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters: ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 07:59:51.930522 29405 raft_consensus.cc:2781] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:51.930621 29405 raft_consensus.cc:686] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:51.930763 29405 consensus_queue.cc:237] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:51.932329 28248 catalog_manager.cc:5484] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 07:59:51.967197 29409 raft_consensus.cc:482] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:51.967325 29409 raft_consensus.cc:504] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.967793 29409 leader_election.cc:288] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:51.968360 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ad2a981fc4c13bc89d22feea7d556" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:51.968559 28399 raft_consensus.cc:2454] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:51.968868 28595 leader_election.cc:302] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:51.969058 29409 raft_consensus.cc:2781] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:51.969115 29409 raft_consensus.cc:482] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:51.969156 29409 raft_consensus.cc:3037] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.970057 29409 raft_consensus.cc:504] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.970450 29409 leader_election.cc:288] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:51.970772 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ad2a981fc4c13bc89d22feea7d556" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:51.970903 28399 raft_consensus.cc:3037] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:51.971768 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ad2a981fc4c13bc89d22feea7d556" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:51.971814 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d08ad2a981fc4c13bc89d22feea7d556" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:51.971887 28399 raft_consensus.cc:2454] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:51.971917 28530 raft_consensus.cc:2454] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:51.972172 28595 leader_election.cc:302] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:51.972330 29409 raft_consensus.cc:2781] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:51.972488 29409 raft_consensus.cc:686] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:51.972638 29409 consensus_queue.cc:237] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:51.974184 28248 catalog_manager.cc:5484] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 07:59:52.009423 29443 raft_consensus.cc:482] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:52.009567 29443 raft_consensus.cc:504] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.010587 29443 leader_election.cc:288] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:52.014309 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bbfb3758e04aeab0df4bbab8380061" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:52.014308 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bbfb3758e04aeab0df4bbab8380061" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:52.014463 28530 raft_consensus.cc:2454] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:52.014468 28661 raft_consensus.cc:2454] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:52.014822 28727 leader_election.cc:302] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:52.015079 29443 raft_consensus.cc:2781] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:52.015139 29443 raft_consensus.cc:482] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:52.015166 29443 raft_consensus.cc:3037] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.016017 29443 raft_consensus.cc:504] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.016393 29443 leader_election.cc:288] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:52.016770 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bbfb3758e04aeab0df4bbab8380061" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:52.016795 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14bbfb3758e04aeab0df4bbab8380061" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:52.016875 28661 raft_consensus.cc:3037] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.016913 28530 raft_consensus.cc:3037] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.017652 28661 raft_consensus.cc:2454] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:52.017678 28530 raft_consensus.cc:2454] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:52.017935 28727 leader_election.cc:302] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:52.018123 29443 raft_consensus.cc:2781] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:52.018324 29443 raft_consensus.cc:686] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 07:59:52.018610 29443 consensus_queue.cc:237] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.020579 28248 catalog_manager.cc:5484] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 07:59:52.150322 29488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:52.150664 29488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:52.154094 29488 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 07:59:52.242276 28530 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.242529 28661 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.243036 29486 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.243202 29443 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.246390 28792 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.246568 28660 raft_consensus.cc:3037] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.247134 28399 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.247406 29405 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.247761 29392 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.248169 28660 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.248714 28529 raft_consensus.cc:3037] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.249748 28529 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.250946 29392 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.251879 29393 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.294754 29445 mvcc.cc:205] Tried to move back new op lower bound from 7023152301035724800 to 7023152300110491648. Current Snapshot: MvccSnapshot[applied={T|T < 7023152301035724800 or (T in {7023152301035724800})}]
INSERT report
rows total: 2000
time total: 105.496 ms
time per row: 0.0527481 ms
I20240502 07:59:52.311879 28256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:52.318737 28771 tablet_service.cc:1467] Processing CreateTablet for tablet 1631d3e38209497ea2454fdfc555a319 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.318696 28378 tablet_service.cc:1467] Processing CreateTablet for tablet 1631d3e38209497ea2454fdfc555a319 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.319051 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1631d3e38209497ea2454fdfc555a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.319051 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1631d3e38209497ea2454fdfc555a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.319326 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 1631d3e38209497ea2454fdfc555a319 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.319553 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1631d3e38209497ea2454fdfc555a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.321360 29541 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:52.321554 28510 tablet_service.cc:1467] Processing CreateTablet for tablet 3288b438832e488e8257227cb8934565 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.321820 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3288b438832e488e8257227cb8934565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.322321 29541 tablet_bootstrap.cc:654] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.322968 29542 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:52.323509 29541 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:52.323662 29541 ts_tablet_manager.cc:1367] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:52.323943 29542 tablet_bootstrap.cc:654] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.324182 29541 raft_consensus.cc:348] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.324362 29541 raft_consensus.cc:374] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.324409 29541 raft_consensus.cc:729] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.324182 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 3288b438832e488e8257227cb8934565 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.324594 29541 consensus_queue.cc:260] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.324765 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3288b438832e488e8257227cb8934565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.324956 29542 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:52.324983 29541 ts_tablet_manager.cc:1398] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:52.325035 29542 ts_tablet_manager.cc:1367] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:52.325424 29542 raft_consensus.cc:348] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.325523 29542 raft_consensus.cc:374] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.325552 29542 raft_consensus.cc:729] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.325678 29542 consensus_queue.cc:260] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.325966 29542 ts_tablet_manager.cc:1398] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.326143 28771 tablet_service.cc:1467] Processing CreateTablet for tablet c47d608895f94a2c82e695ca16677abf (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.326400 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47d608895f94a2c82e695ca16677abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.326658 28377 tablet_service.cc:1467] Processing CreateTablet for tablet 3288b438832e488e8257227cb8934565 (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.326901 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3288b438832e488e8257227cb8934565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.327059 29541 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:52.327994 29541 tablet_bootstrap.cc:654] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.328985 29542 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:52.329174 29541 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:52.329246 29541 ts_tablet_manager.cc:1367] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:52.329243 28639 tablet_service.cc:1467] Processing CreateTablet for tablet c47d608895f94a2c82e695ca16677abf (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.329517 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47d608895f94a2c82e695ca16677abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.329676 29541 raft_consensus.cc:348] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.329792 29541 raft_consensus.cc:374] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.329825 29541 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.329973 29541 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.330061 28378 tablet_service.cc:1467] Processing CreateTablet for tablet c47d608895f94a2c82e695ca16677abf (DEFAULT_TABLE table=rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.330286 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47d608895f94a2c82e695ca16677abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.330322 29542 tablet_bootstrap.cc:654] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.330327 29541 ts_tablet_manager.cc:1398] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:52.331585 29542 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:52.331660 29542 ts_tablet_manager.cc:1367] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:52.332134 29542 raft_consensus.cc:348] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.332237 29542 raft_consensus.cc:374] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.332271 29542 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.332552 29542 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.332835 29548 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:52.332906 29542 ts_tablet_manager.cc:1398] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:52.333894 29543 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:52.335601 29543 tablet_bootstrap.cc:654] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.335604 29548 tablet_bootstrap.cc:654] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.336974 29541 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:52.337061 29542 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:52.337213 29548 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:52.337282 29548 ts_tablet_manager.cc:1367] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:52.337635 29548 raft_consensus.cc:348] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.337716 29548 raft_consensus.cc:374] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.337738 29548 raft_consensus.cc:729] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.337852 29548 consensus_queue.cc:260] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.337975 29542 tablet_bootstrap.cc:654] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.337967 29541 tablet_bootstrap.cc:654] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.338104 29548 ts_tablet_manager.cc:1398] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:52.339481 29541 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:52.339537 29543 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:52.339543 29542 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:52.339573 29541 ts_tablet_manager.cc:1367] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:52.339618 29543 ts_tablet_manager.cc:1367] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:59:52.339634 29542 ts_tablet_manager.cc:1367] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:52.340030 29543 raft_consensus.cc:348] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.340030 29541 raft_consensus.cc:348] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.340137 29543 raft_consensus.cc:374] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.340137 29541 raft_consensus.cc:374] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.340170 29541 raft_consensus.cc:729] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.340170 29543 raft_consensus.cc:729] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.340101 29542 raft_consensus.cc:348] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.340224 29542 raft_consensus.cc:374] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.340261 29542 raft_consensus.cc:729] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.340301 29543 consensus_queue.cc:260] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.340294 29541 consensus_queue.cc:260] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.340374 29542 consensus_queue.cc:260] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.340584 29541 ts_tablet_manager.cc:1398] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:52.340584 29543 ts_tablet_manager.cc:1398] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.340709 29542 ts_tablet_manager.cc:1398] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:52.340759 29543 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:52.341661 29543 tablet_bootstrap.cc:654] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.342922 29543 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:52.343027 29543 ts_tablet_manager.cc:1367] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:52.343498 29543 raft_consensus.cc:348] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.343616 29543 raft_consensus.cc:374] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.343649 29543 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.343777 29543 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.344084 29543 ts_tablet_manager.cc:1398] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:52.469517 29390 raft_consensus.cc:482] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:52.469666 29390 raft_consensus.cc:504] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.470185 29390 leader_election.cc:288] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:52.470824 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47d608895f94a2c82e695ca16677abf" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:52.470993 28661 raft_consensus.cc:2454] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:52.471408 28334 leader_election.cc:302] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:52.471632 29390 raft_consensus.cc:2781] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:52.471705 29390 raft_consensus.cc:482] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:52.471761 29390 raft_consensus.cc:3037] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.472823 29390 raft_consensus.cc:504] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.473230 29390 leader_election.cc:288] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:52.473608 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47d608895f94a2c82e695ca16677abf" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:52.473719 28661 raft_consensus.cc:3037] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.474735 28661 raft_consensus.cc:2454] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:52.475072 28334 leader_election.cc:302] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:52.475100 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47d608895f94a2c82e695ca16677abf" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:52.475194 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c47d608895f94a2c82e695ca16677abf" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:52.475252 29390 raft_consensus.cc:2781] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:52.475234 28792 raft_consensus.cc:2454] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:52.475430 29390 raft_consensus.cc:686] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 07:59:52.475595 29390 consensus_queue.cc:237] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:52.477375 28257 catalog_manager.cc:5484] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 07:59:52.489357 29390 raft_consensus.cc:482] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:52.489496 29390 raft_consensus.cc:504] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.489929 29390 leader_election.cc:288] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:52.490408 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3288b438832e488e8257227cb8934565" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:52.490435 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3288b438832e488e8257227cb8934565" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:52.490554 28529 raft_consensus.cc:2454] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:52.490582 28791 raft_consensus.cc:2454] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:52.490904 28332 leader_election.cc:302] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:52.491079 29390 raft_consensus.cc:2781] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:52.491137 29390 raft_consensus.cc:482] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:52.491164 29390 raft_consensus.cc:3037] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.491986 29390 raft_consensus.cc:504] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.492367 29390 leader_election.cc:288] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:52.492774 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3288b438832e488e8257227cb8934565" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:52.492769 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3288b438832e488e8257227cb8934565" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:52.492885 28529 raft_consensus.cc:3037] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.492909 28791 raft_consensus.cc:3037] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.493640 28529 raft_consensus.cc:2454] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:52.493773 28791 raft_consensus.cc:2454] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:52.493947 28332 leader_election.cc:302] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:52.494129 29390 raft_consensus.cc:2781] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:52.494210 29390 raft_consensus.cc:686] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 07:59:52.494343 29390 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.496134 28257 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 07:59:52.642045 29390 raft_consensus.cc:482] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:52.642220 29390 raft_consensus.cc:504] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.642769 29390 leader_election.cc:288] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:52.643260 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:52.643293 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:52.643445 28791 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:52.643458 28529 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:52.643795 28335 leader_election.cc:302] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:52.644002 29390 raft_consensus.cc:2781] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:52.644065 29390 raft_consensus.cc:482] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:52.644093 29390 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.644933 29390 raft_consensus.cc:504] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.645285 29390 leader_election.cc:288] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:52.645779 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:52.645908 28529 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.646695 28529 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:52.646984 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:52.647043 28332 leader_election.cc:302] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:52.647136 28791 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.647254 29390 raft_consensus.cc:2781] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:52.647359 29390 raft_consensus.cc:686] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 07:59:52.647518 29390 consensus_queue.cc:237] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.648058 28791 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:52.649349 28257 catalog_manager.cc:5484] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
W20240502 07:59:52.808357 29559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:52.808676 29559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:52.811815 29559 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 07:59:52.857733 28791 raft_consensus.cc:3037] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:52.859508 29390 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.871073 29555 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.894598 28529 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.894990 28791 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.895740 29390 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.896031 29390 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.906049 28530 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.906563 29555 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:52.907119 28792 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:52.909456 29555 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 94.5332 ms
time per row: 0.0472666 ms
I20240502 07:59:52.955121 28256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:52.961478 28639 tablet_service.cc:1467] Processing CreateTablet for tablet 167b5f3b818048af81b1350839ebe489 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.961652 28641 tablet_service.cc:1467] Processing CreateTablet for tablet df20642aef7748ff8962b3095e1ab56c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.961870 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167b5f3b818048af81b1350839ebe489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.962026 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20642aef7748ff8962b3095e1ab56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.962217 28378 tablet_service.cc:1467] Processing CreateTablet for tablet 167b5f3b818048af81b1350839ebe489 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.962038 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 21d939e5035541a4b003c5d276493d5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.962472 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.962491 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167b5f3b818048af81b1350839ebe489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.964890 29614 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:52.965024 28510 tablet_service.cc:1467] Processing CreateTablet for tablet 167b5f3b818048af81b1350839ebe489 (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.965209 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 21d939e5035541a4b003c5d276493d5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.965296 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167b5f3b818048af81b1350839ebe489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.965402 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.965710 29614 tablet_bootstrap.cc:654] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.966696 29614 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:52.966759 29614 ts_tablet_manager.cc:1367] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:52.967113 29614 raft_consensus.cc:348] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.967218 29614 raft_consensus.cc:374] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.967253 29614 raft_consensus.cc:729] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.967411 29614 consensus_queue.cc:260] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.967636 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 21d939e5035541a4b003c5d276493d5c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.967959 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.968173 29616 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:52.968274 29614 ts_tablet_manager.cc:1398] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.969183 29616 tablet_bootstrap.cc:654] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.967636 28771 tablet_service.cc:1467] Processing CreateTablet for tablet df20642aef7748ff8962b3095e1ab56c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.969565 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20642aef7748ff8962b3095e1ab56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.970471 29616 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:52.970563 29616 ts_tablet_manager.cc:1367] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:52.970749 29618 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:52.971032 29616 raft_consensus.cc:348] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.971141 29616 raft_consensus.cc:374] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.971171 29616 raft_consensus.cc:729] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.971310 29616 consensus_queue.cc:260] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.971612 29616 ts_tablet_manager.cc:1398] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.971676 29618 tablet_bootstrap.cc:654] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.971800 29616 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:52.972410 29613 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:52.972739 28378 tablet_service.cc:1467] Processing CreateTablet for tablet df20642aef7748ff8962b3095e1ab56c (DEFAULT_TABLE table=rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:52.973017 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20642aef7748ff8962b3095e1ab56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:52.973035 29618 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:52.973066 29616 tablet_bootstrap.cc:654] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.973124 29618 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:52.973598 29613 tablet_bootstrap.cc:654] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.973618 29618 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.973754 29618 raft_consensus.cc:374] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.973803 29618 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.973955 29618 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.974256 29618 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.974313 29616 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:52.974390 29616 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:52.974437 29618 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:52.974850 29616 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.974995 29616 raft_consensus.cc:374] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.975026 29616 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.975133 29614 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:52.975150 29616 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.975360 29618 tablet_bootstrap.cc:654] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.975441 29616 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.976019 29614 tablet_bootstrap.cc:654] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.976912 29618 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:52.976994 29618 ts_tablet_manager.cc:1367] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:52.977116 29614 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:52.977212 29614 ts_tablet_manager.cc:1367] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:52.977427 29618 raft_consensus.cc:348] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.977550 29618 raft_consensus.cc:374] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.977588 29618 raft_consensus.cc:729] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.977720 29613 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:52.977706 29618 consensus_queue.cc:260] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.977895 29613 ts_tablet_manager.cc:1367] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 07:59:52.977941 29614 raft_consensus.cc:348] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.978034 29618 ts_tablet_manager.cc:1398] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.978058 29614 raft_consensus.cc:374] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.978096 29614 raft_consensus.cc:729] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.978222 29614 consensus_queue.cc:260] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.978478 29614 ts_tablet_manager.cc:1398] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.978482 29613 raft_consensus.cc:348] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.978570 29613 raft_consensus.cc:374] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.978605 29613 raft_consensus.cc:729] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.978717 29613 consensus_queue.cc:260] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:52.979490 29613 ts_tablet_manager.cc:1398] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:52.979696 29613 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:52.980692 29613 tablet_bootstrap.cc:654] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.982067 29613 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:52.982158 29613 ts_tablet_manager.cc:1367] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:52.982519 29613 raft_consensus.cc:348] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.982635 29613 raft_consensus.cc:374] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.982681 29613 raft_consensus.cc:729] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.982816 29613 consensus_queue.cc:260] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:52.983059 29613 ts_tablet_manager.cc:1398] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:52.983268 29613 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:52.983989 29613 tablet_bootstrap.cc:654] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:52.985059 29613 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:52.985188 29613 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:52.985594 29613 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.985708 29613 raft_consensus.cc:374] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:52.985745 29613 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:52.985852 29613 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:52.986091 29613 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:53.054450 29443 raft_consensus.cc:482] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:53.054616 29443 raft_consensus.cc:504] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.055085 29443 leader_election.cc:288] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:53.055541 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d939e5035541a4b003c5d276493d5c" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:53.055542 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d939e5035541a4b003c5d276493d5c" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:53.055682 28530 raft_consensus.cc:2454] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:53.055697 28661 raft_consensus.cc:2454] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:53.056025 28727 leader_election.cc:302] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.056217 29443 raft_consensus.cc:2781] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:53.056277 29443 raft_consensus.cc:482] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:53.056303 29443 raft_consensus.cc:3037] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.057197 29443 raft_consensus.cc:504] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.057561 29443 leader_election.cc:288] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:53.057956 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d939e5035541a4b003c5d276493d5c" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:53.058004 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21d939e5035541a4b003c5d276493d5c" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:53.058079 28530 raft_consensus.cc:3037] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.058104 28661 raft_consensus.cc:3037] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.058910 28661 raft_consensus.cc:2454] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:53.058984 28530 raft_consensus.cc:2454] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:53.059360 28725 leader_election.cc:302] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:53.059535 29443 raft_consensus.cc:2781] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:53.059762 29443 raft_consensus.cc:686] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 07:59:53.059921 29443 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.061661 28248 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 07:59:53.186820 29405 raft_consensus.cc:482] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:53.186959 29405 raft_consensus.cc:504] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.187440 29405 leader_election.cc:288] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:53.187950 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167b5f3b818048af81b1350839ebe489" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:53.187997 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167b5f3b818048af81b1350839ebe489" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:53.188110 28661 raft_consensus.cc:2454] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:53.188155 28399 raft_consensus.cc:2454] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:53.188515 28465 leader_election.cc:302] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.188720 29405 raft_consensus.cc:2781] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:53.188782 29405 raft_consensus.cc:482] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:53.188817 29405 raft_consensus.cc:3037] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.189843 29405 raft_consensus.cc:504] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.190209 29405 leader_election.cc:288] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:53.190598 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167b5f3b818048af81b1350839ebe489" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:53.190637 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167b5f3b818048af81b1350839ebe489" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:53.190739 28661 raft_consensus.cc:3037] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.190749 28399 raft_consensus.cc:3037] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.191529 28661 raft_consensus.cc:2454] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:53.191686 28399 raft_consensus.cc:2454] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:53.191896 28465 leader_election.cc:302] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.192075 29405 raft_consensus.cc:2781] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:53.192282 29405 raft_consensus.cc:686] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:53.192449 29405 consensus_queue.cc:237] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.194190 28248 catalog_manager.cc:5484] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 07:59:53.261322 29392 raft_consensus.cc:482] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:53.261457 29392 raft_consensus.cc:504] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:53.261853 29392 leader_election.cc:288] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:53.262285 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20642aef7748ff8962b3095e1ab56c" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:53.262285 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20642aef7748ff8962b3095e1ab56c" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:53.262436 28399 raft_consensus.cc:2454] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:53.262435 28791 raft_consensus.cc:2454] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:53.262765 28595 leader_election.cc:302] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.262949 29392 raft_consensus.cc:2781] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:53.263013 29392 raft_consensus.cc:482] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:53.263042 29392 raft_consensus.cc:3037] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.263909 29392 raft_consensus.cc:504] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:53.264289 29392 leader_election.cc:288] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:53.264705 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20642aef7748ff8962b3095e1ab56c" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:53.264818 28791 raft_consensus.cc:3037] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.264801 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df20642aef7748ff8962b3095e1ab56c" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:53.264946 28399 raft_consensus.cc:3037] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.265615 28791 raft_consensus.cc:2454] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:53.265769 28399 raft_consensus.cc:2454] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:53.265937 28597 leader_election.cc:302] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.266104 29392 raft_consensus.cc:2781] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:53.266322 29392 raft_consensus.cc:686] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:53.266466 29392 consensus_queue.cc:237] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:53.268168 28248 catalog_manager.cc:5484] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
W20240502 07:59:53.444494 29632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:53.444783 29632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:53.448145 29632 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 07:59:53.508679 29443 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:53.512478 29626 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:53.538671 28399 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:53.539112 29405 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:53.539572 28661 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:53.539976 29405 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:53.540663 28791 raft_consensus.cc:1261] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:53.540887 28399 raft_consensus.cc:1261] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:53.541141 29630 consensus_queue.cc:1035] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:53.542968 29392 consensus_queue.cc:1035] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 115.462 ms
time per row: 0.0577312 ms
I20240502 07:59:53.613034 28253 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:53.619071 28510 tablet_service.cc:1467] Processing CreateTablet for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.619134 28378 tablet_service.cc:1467] Processing CreateTablet for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.619451 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.619447 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.619423 28771 tablet_service.cc:1467] Processing CreateTablet for tablet fddc382ab3164ccdaa86f802c83da8be (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.619676 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddc382ab3164ccdaa86f802c83da8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.619957 28377 tablet_service.cc:1467] Processing CreateTablet for tablet fddc382ab3164ccdaa86f802c83da8be (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.620163 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddc382ab3164ccdaa86f802c83da8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.622043 28379 tablet_service.cc:1467] Processing CreateTablet for tablet f98044a1730e4a04a7088199d4f0850e (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.622319 28379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98044a1730e4a04a7088199d4f0850e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.622699 29697 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:53.622740 28771 tablet_service.cc:1467] Processing CreateTablet for tablet f98044a1730e4a04a7088199d4f0850e (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.622854 28640 tablet_service.cc:1467] Processing CreateTablet for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.622989 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98044a1730e4a04a7088199d4f0850e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.623124 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.623919 29697 tablet_bootstrap.cc:654] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.624509 28510 tablet_service.cc:1467] Processing CreateTablet for tablet fddc382ab3164ccdaa86f802c83da8be (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.624759 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddc382ab3164ccdaa86f802c83da8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.624969 29696 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:53.625124 29695 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:53.625398 29699 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:53.625566 28641 tablet_service.cc:1467] Processing CreateTablet for tablet f98044a1730e4a04a7088199d4f0850e (DEFAULT_TABLE table=rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:53.625802 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98044a1730e4a04a7088199d4f0850e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:53.626323 29699 tablet_bootstrap.cc:654] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.626967 29695 tablet_bootstrap.cc:654] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.627213 29696 tablet_bootstrap.cc:654] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.627530 29699 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:53.627640 29699 ts_tablet_manager.cc:1367] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:53.628151 29699 raft_consensus.cc:348] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.628223 29695 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:53.628283 29699 raft_consensus.cc:374] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.628319 29695 ts_tablet_manager.cc:1367] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:53.628317 29699 raft_consensus.cc:729] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.628523 29699 consensus_queue.cc:260] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.628855 29699 ts_tablet_manager.cc:1398] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:53.628830 29695 raft_consensus.cc:348] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.628935 29695 raft_consensus.cc:374] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.628975 29695 raft_consensus.cc:729] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.629036 29699 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:53.629117 29695 consensus_queue.cc:260] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.629385 29695 ts_tablet_manager.cc:1398] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:53.629552 29695 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:53.629968 29699 tablet_bootstrap.cc:654] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.630599 29695 tablet_bootstrap.cc:654] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.631512 29696 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:53.631637 29696 ts_tablet_manager.cc:1367] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:53.631978 29695 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:53.631978 29699 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:53.632077 29699 ts_tablet_manager.cc:1367] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:53.632166 29695 ts_tablet_manager.cc:1367] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:53.632144 29696 raft_consensus.cc:348] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.632278 29696 raft_consensus.cc:374] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.632324 29696 raft_consensus.cc:729] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.632501 29696 consensus_queue.cc:260] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.632776 29696 ts_tablet_manager.cc:1398] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:53.632966 29696 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:53.633389 29699 raft_consensus.cc:348] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.633527 29699 raft_consensus.cc:374] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.633569 29699 raft_consensus.cc:729] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.633781 29695 raft_consensus.cc:348] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.633961 29696 tablet_bootstrap.cc:654] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.633960 29695 raft_consensus.cc:374] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.634083 29695 raft_consensus.cc:729] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.634307 29695 consensus_queue.cc:260] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.634202 29699 consensus_queue.cc:260] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.634748 29695 ts_tablet_manager.cc:1398] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:53.634855 29699 ts_tablet_manager.cc:1398] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:53.635020 29695 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:53.635514 29697 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:53.635627 29697 ts_tablet_manager.cc:1367] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240502 07:59:53.636142 29695 tablet_bootstrap.cc:654] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.636126 29697 raft_consensus.cc:348] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.636258 29697 raft_consensus.cc:374] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.636289 29697 raft_consensus.cc:729] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.636404 29697 consensus_queue.cc:260] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.636765 29697 ts_tablet_manager.cc:1398] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:53.636940 29697 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:53.637219 29696 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:53.637329 29696 ts_tablet_manager.cc:1367] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:53.637801 29696 raft_consensus.cc:348] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.637887 29697 tablet_bootstrap.cc:654] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:53.637940 29696 raft_consensus.cc:374] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.637987 29696 raft_consensus.cc:729] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.638104 29696 consensus_queue.cc:260] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.638422 29696 ts_tablet_manager.cc:1398] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:53.640138 29695 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:53.640218 29697 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:53.640246 29695 ts_tablet_manager.cc:1367] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 07:59:53.640312 29697 ts_tablet_manager.cc:1367] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:53.640719 29695 raft_consensus.cc:348] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.640834 29695 raft_consensus.cc:374] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.640869 29695 raft_consensus.cc:729] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.640816 29697 raft_consensus.cc:348] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.640919 29697 raft_consensus.cc:374] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:53.640959 29697 raft_consensus.cc:729] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:53.641008 29695 consensus_queue.cc:260] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.641095 29697 consensus_queue.cc:260] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.641302 29695 ts_tablet_manager.cc:1398] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:53.641396 29697 ts_tablet_manager.cc:1398] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:53.646420 29692 raft_consensus.cc:482] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:53.646545 29692 raft_consensus.cc:504] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.647054 29692 leader_election.cc:288] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:53.647411 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:53.647464 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:53.647542 28530 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:53.647629 28398 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:53.647872 28594 leader_election.cc:302] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.648195 29692 raft_consensus.cc:2781] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:53.648264 29692 raft_consensus.cc:482] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:53.648311 29692 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.649266 29692 raft_consensus.cc:504] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.649624 29692 leader_election.cc:288] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:53.649935 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:53.650076 28530 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.650909 28530 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:53.651160 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:53.651217 28594 leader_election.cc:302] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.651285 28398 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.651396 29692 raft_consensus.cc:2781] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:53.651479 29692 raft_consensus.cc:686] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:53.651643 29692 consensus_queue.cc:237] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:53.652146 28398 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:53.653483 28254 catalog_manager.cc:5484] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 07:59:53.656231 29692 raft_consensus.cc:482] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:53.656322 29692 raft_consensus.cc:504] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.656730 29692 leader_election.cc:288] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:53.657157 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98044a1730e4a04a7088199d4f0850e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:53.657285 28791 raft_consensus.cc:2454] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:53.657539 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98044a1730e4a04a7088199d4f0850e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:53.657593 28597 leader_election.cc:302] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.657675 28398 raft_consensus.cc:2454] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:53.657758 29692 raft_consensus.cc:2781] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:53.657811 29692 raft_consensus.cc:482] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:53.657871 29692 raft_consensus.cc:3037] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.658735 29692 raft_consensus.cc:504] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.659159 29692 leader_election.cc:288] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:53.659459 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98044a1730e4a04a7088199d4f0850e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:53.659564 28398 raft_consensus.cc:3037] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.659543 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f98044a1730e4a04a7088199d4f0850e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:53.659688 28791 raft_consensus.cc:3037] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.660431 28398 raft_consensus.cc:2454] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:53.660624 28791 raft_consensus.cc:2454] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:53.660923 28595 leader_election.cc:302] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:53.661214 29692 raft_consensus.cc:2781] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:53.661314 29692 raft_consensus.cc:686] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:53.661458 29692 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:53.663123 28254 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 07:59:53.748872 29679 raft_consensus.cc:482] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:53.749025 29679 raft_consensus.cc:504] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.749711 29679 leader_election.cc:288] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:53.750288 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddc382ab3164ccdaa86f802c83da8be" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:53.750448 28530 raft_consensus.cc:2454] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:53.750814 28725 leader_election.cc:302] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:53.751008 29679 raft_consensus.cc:2781] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:53.751053 29679 raft_consensus.cc:482] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:53.751088 29679 raft_consensus.cc:3037] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.752493 29679 raft_consensus.cc:504] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.752851 29679 leader_election.cc:288] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:53.753353 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddc382ab3164ccdaa86f802c83da8be" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:53.753474 28530 raft_consensus.cc:3037] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.754096 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddc382ab3164ccdaa86f802c83da8be" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:53.754237 28398 raft_consensus.cc:2454] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:53.754402 28530 raft_consensus.cc:2454] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:53.754405 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fddc382ab3164ccdaa86f802c83da8be" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:53.754498 28397 raft_consensus.cc:3037] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:53.754753 28725 leader_election.cc:302] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:53.754987 29679 raft_consensus.cc:2781] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:53.755151 29679 raft_consensus.cc:686] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 07:59:53.755297 29679 consensus_queue.cc:237] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:53.755939 28397 raft_consensus.cc:2454] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:53.757061 28254 catalog_manager.cc:5484] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
W20240502 07:59:53.903185 29713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:53.903468 29713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:53.906653 29713 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 07:59:54.011384 28397 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.011385 28530 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.012033 28791 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.012435 29692 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.012850 29692 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.013499 29630 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.017170 28398 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.017984 29679 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.018973 28530 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.019676 29679 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.027335 28399 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.032435 29392 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 141.276 ms
time per row: 0.0706378 ms
I20240502 07:59:54.096995 28254 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:54.103420 28510 tablet_service.cc:1467] Processing CreateTablet for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.103495 28771 tablet_service.cc:1467] Processing CreateTablet for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.103462 28641 tablet_service.cc:1467] Processing CreateTablet for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.103770 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d358245cdf4b2ebb0b5005f75cf1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.103770 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d358245cdf4b2ebb0b5005f75cf1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.103797 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d358245cdf4b2ebb0b5005f75cf1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.104794 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.104921 28508 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7e7dadff0a4cd08deb043a37a54424 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.105046 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0499a3f3e6bd4069bf992e32f1abfdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.105163 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7e7dadff0a4cd08deb043a37a54424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.106655 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.106936 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0499a3f3e6bd4069bf992e32f1abfdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.107072 29697 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:54.107568 29696 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:54.107920 29697 tablet_bootstrap.cc:654] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.108510 29696 tablet_bootstrap.cc:654] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.108805 28379 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7e7dadff0a4cd08deb043a37a54424 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.108906 29699 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:54.109095 28379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7e7dadff0a4cd08deb043a37a54424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.109637 29699 tablet_bootstrap.cc:654] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.109766 29696 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:54.109853 29696 ts_tablet_manager.cc:1367] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:54.110000 28641 tablet_service.cc:1467] Processing CreateTablet for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.110241 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0499a3f3e6bd4069bf992e32f1abfdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.110409 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 9c7e7dadff0a4cd08deb043a37a54424 (DEFAULT_TABLE table=rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.110623 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7e7dadff0a4cd08deb043a37a54424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.110570 29696 raft_consensus.cc:348] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.110677 29696 raft_consensus.cc:374] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.110708 29696 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.110844 29696 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.111128 29696 ts_tablet_manager.cc:1398] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 07:59:54.111302 29695 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:54.112134 29695 tablet_bootstrap.cc:654] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.113180 29695 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:54.113173 29699 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:54.113261 29699 ts_tablet_manager.cc:1367] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:54.113261 29695 ts_tablet_manager.cc:1367] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 07:59:54.113262 29696 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:54.113706 29699 raft_consensus.cc:348] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.113695 29695 raft_consensus.cc:348] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.113811 29699 raft_consensus.cc:374] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.113811 29695 raft_consensus.cc:374] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.113843 29695 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.113843 29699 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.113965 29699 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.113965 29695 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.114243 29695 ts_tablet_manager.cc:1398] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:54.114238 29699 ts_tablet_manager.cc:1398] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.114403 29699 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:54.114679 29697 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:54.114773 29697 ts_tablet_manager.cc:1367] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240502 07:59:54.115234 29697 raft_consensus.cc:348] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.115374 29697 raft_consensus.cc:374] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.115411 29697 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.115533 29697 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.115260 29699 tablet_bootstrap.cc:654] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.115813 29697 ts_tablet_manager.cc:1398] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.116009 29697 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:54.116093 29696 tablet_bootstrap.cc:654] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.116987 29697 tablet_bootstrap.cc:654] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.117538 29696 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:54.117643 29696 ts_tablet_manager.cc:1367] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:54.117790 29699 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:54.117873 29699 ts_tablet_manager.cc:1367] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 07:59:54.118110 29696 raft_consensus.cc:348] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.118228 29696 raft_consensus.cc:374] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.118264 29696 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.118275 29699 raft_consensus.cc:348] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.118376 29699 raft_consensus.cc:374] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.118407 29699 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.118403 29696 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.118512 29699 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.118775 29699 ts_tablet_manager.cc:1398] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.118938 29699 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:54.119290 29696 ts_tablet_manager.cc:1398] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:54.119863 29699 tablet_bootstrap.cc:654] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.120723 29697 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:54.120820 29697 ts_tablet_manager.cc:1367] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 07:59:54.121400 29699 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:54.121357 29697 raft_consensus.cc:348] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.121470 29697 raft_consensus.cc:374] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.121503 29697 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.121505 29699 ts_tablet_manager.cc:1367] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:54.121618 29697 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.121902 29697 ts_tablet_manager.cc:1398] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:54.121989 29699 raft_consensus.cc:348] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.122098 29697 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:54.122120 29699 raft_consensus.cc:374] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.122160 29699 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.122314 29699 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.122661 29699 ts_tablet_manager.cc:1398] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.123487 29697 tablet_bootstrap.cc:654] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.124816 29697 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:54.124933 29697 ts_tablet_manager.cc:1367] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:54.125504 29697 raft_consensus.cc:348] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.125622 29697 raft_consensus.cc:374] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.125648 29697 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.125777 29697 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.126390 29697 ts_tablet_manager.cc:1398] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.148563 29628 raft_consensus.cc:482] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:54.148728 29628 raft_consensus.cc:504] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.149224 29628 leader_election.cc:288] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:54.149703 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:54.149632 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:54.149855 28397 raft_consensus.cc:2454] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.149855 28661 raft_consensus.cc:2454] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.150183 28464 leader_election.cc:302] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:54.150403 29628 raft_consensus.cc:2781] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:54.150476 29628 raft_consensus.cc:482] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:54.150511 29628 raft_consensus.cc:3037] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.151472 29628 raft_consensus.cc:504] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.151834 29628 leader_election.cc:288] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:54.152200 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:54.152202 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:54.152335 28397 raft_consensus.cc:3037] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.152345 28661 raft_consensus.cc:3037] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.153169 28661 raft_consensus.cc:2454] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.153172 28397 raft_consensus.cc:2454] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.153514 28464 leader_election.cc:302] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:54.153687 29628 raft_consensus.cc:2781] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:54.153828 29628 raft_consensus.cc:686] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:54.153987 29628 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.155877 28253 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 07:59:54.191154 29628 raft_consensus.cc:482] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:54.191308 29628 raft_consensus.cc:504] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.191830 29628 leader_election.cc:288] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.192369 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:54.192381 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:54.192553 28791 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.192561 28661 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.192896 28465 leader_election.cc:302] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:54.193080 29628 raft_consensus.cc:2781] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:54.193142 29628 raft_consensus.cc:482] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:54.193214 29628 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.217382 29628 raft_consensus.cc:504] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.217907 29628 leader_election.cc:288] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.218401 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:54.218431 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:54.218539 28791 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.218591 28661 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.219519 28791 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.219548 28661 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.219931 28466 leader_election.cc:302] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:54.220170 29628 raft_consensus.cc:2781] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:54.220278 29628 raft_consensus.cc:686] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:54.220427 29628 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.222424 28253 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 07:59:54.391625 29628 raft_consensus.cc:482] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:54.391767 29628 raft_consensus.cc:504] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.392236 29628 leader_election.cc:288] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.392741 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:54.392755 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:54.392892 28791 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.392903 28661 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.393220 28466 leader_election.cc:302] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:54.393409 29628 raft_consensus.cc:2781] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:54.393469 29628 raft_consensus.cc:482] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:54.393527 29628 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.394348 29628 raft_consensus.cc:504] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.394718 29628 leader_election.cc:288] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.395076 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:54.395076 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:54.395202 28661 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.395211 28791 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.396142 28661 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.396142 28791 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.396500 28465 leader_election.cc:302] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:54.396699 29628 raft_consensus.cc:2781] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:54.396819 29628 raft_consensus.cc:686] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:54.396970 29628 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.398890 28253 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 07:59:54.527184 29758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:54.527477 29758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:54.530818 29758 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 07:59:54.612385 28661 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.612931 29405 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.613083 28661 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.613631 29405 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.613875 28791 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.614220 28792 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.616000 29405 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.616315 29405 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.621335 28397 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.621950 29628 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.623287 28659 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:54.623741 29628 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:54.638340 29785 mvcc.cc:205] Tried to move back new op lower bound from 7023152310728347648 to 7023152309128581120. Current Snapshot: MvccSnapshot[applied={T|T < 7023152310728347648}]
INSERT report
rows total: 2000
time total: 104.847 ms
time per row: 0.0524234 ms
I20240502 07:59:54.682417 28248 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:54.688385 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 6a40382745f84f0eadf9e0ee2e057289 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.688445 28379 tablet_service.cc:1467] Processing CreateTablet for tablet 6a40382745f84f0eadf9e0ee2e057289 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.688776 28379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a40382745f84f0eadf9e0ee2e057289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.688824 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a40382745f84f0eadf9e0ee2e057289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.689808 28378 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1c1fa046b741aa8cf296aa57847fe3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.689774 28377 tablet_service.cc:1467] Processing CreateTablet for tablet 698f494a22d34e659766b97cb24d0293 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.689853 28508 tablet_service.cc:1467] Processing CreateTablet for tablet 6a40382745f84f0eadf9e0ee2e057289 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.690075 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1c1fa046b741aa8cf296aa57847fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.690148 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a40382745f84f0eadf9e0ee2e057289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.690203 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.690573 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1c1fa046b741aa8cf296aa57847fe3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.690774 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1c1fa046b741aa8cf296aa57847fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.692286 29815 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:54.692441 28770 tablet_service.cc:1467] Processing CreateTablet for tablet 5d1c1fa046b741aa8cf296aa57847fe3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.692438 28771 tablet_service.cc:1467] Processing CreateTablet for tablet 698f494a22d34e659766b97cb24d0293 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.692807 28770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1c1fa046b741aa8cf296aa57847fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.692982 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.693215 29815 tablet_bootstrap.cc:654] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.693825 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 698f494a22d34e659766b97cb24d0293 (DEFAULT_TABLE table=rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:54.694078 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:54.694272 29815 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:54.694334 29815 ts_tablet_manager.cc:1367] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:54.694679 29815 raft_consensus.cc:348] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.694763 29815 raft_consensus.cc:374] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.694784 29815 raft_consensus.cc:729] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.694885 29815 consensus_queue.cc:260] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.695122 29815 ts_tablet_manager.cc:1398] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:54.695277 29815 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:54.696178 29815 tablet_bootstrap.cc:654] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.696338 29817 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:54.697247 29817 tablet_bootstrap.cc:654] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.697377 29815 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:54.697468 29815 ts_tablet_manager.cc:1367] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:54.698002 29815 raft_consensus.cc:348] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.698130 29815 raft_consensus.cc:374] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.698189 29815 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.698333 29815 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.698801 29815 ts_tablet_manager.cc:1398] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.699542 29817 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:54.699637 29817 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:54.700050 29817 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.700176 29817 raft_consensus.cc:374] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.700209 29817 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.700345 29817 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.700659 29817 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.710928 29821 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:54.711683 29820 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:54.711942 29821 tablet_bootstrap.cc:654] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.712857 29820 tablet_bootstrap.cc:654] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.713441 29821 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:54.713589 29821 ts_tablet_manager.cc:1367] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:54.713900 29820 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:54.713987 29820 ts_tablet_manager.cc:1367] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:54.713966 29821 raft_consensus.cc:348] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.714087 29821 raft_consensus.cc:374] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.714131 29821 raft_consensus.cc:729] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.714274 29821 consensus_queue.cc:260] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.714449 29820 raft_consensus.cc:348] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.714545 29821 ts_tablet_manager.cc:1398] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:54.714557 29820 raft_consensus.cc:374] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.714591 29820 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.714715 29821 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:54.714711 29820 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.715224 29820 ts_tablet_manager.cc:1398] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.715397 29820 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:54.716327 29820 tablet_bootstrap.cc:654] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.716347 29821 tablet_bootstrap.cc:654] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.717695 29820 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:54.717720 29821 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:54.717779 29820 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:54.717810 29821 ts_tablet_manager.cc:1367] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:54.718173 29820 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.718291 29820 raft_consensus.cc:374] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.718328 29820 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.718319 29821 raft_consensus.cc:348] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.718434 29821 raft_consensus.cc:374] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.718473 29821 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.718456 29820 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.718611 29821 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.718716 29820 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.718871 29820 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:54.718909 29821 ts_tablet_manager.cc:1398] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.719086 29821 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:54.719983 29820 tablet_bootstrap.cc:654] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.720203 29821 tablet_bootstrap.cc:654] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:54.721314 29821 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:54.721314 29820 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:54.721396 29820 ts_tablet_manager.cc:1367] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:54.721396 29821 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:54.721810 29821 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.721808 29820 raft_consensus.cc:348] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.721904 29821 raft_consensus.cc:374] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.721904 29820 raft_consensus.cc:374] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:54.721937 29821 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.721937 29820 raft_consensus.cc:729] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:54.722056 29821 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.722077 29820 consensus_queue.cc:260] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.722371 29821 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.722441 29820 ts_tablet_manager.cc:1398] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:54.725693 29798 raft_consensus.cc:482] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:54.725809 29798 raft_consensus.cc:504] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.726171 29798 leader_election.cc:288] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.726547 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:54.726675 28791 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.727030 28466 leader_election.cc:302] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:54.727257 29798 raft_consensus.cc:2781] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:54.727268 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:54.727319 29798 raft_consensus.cc:482] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:54.727370 29798 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.727394 28397 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:54.728346 29798 raft_consensus.cc:504] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.728739 29798 leader_election.cc:288] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.729039 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:54.729089 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:54.729172 28397 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.729208 28791 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.729991 28397 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.730168 28791 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:54.730286 28464 leader_election.cc:302] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 07:59:54.730446 29798 raft_consensus.cc:2781] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:54.730571 29798 raft_consensus.cc:686] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:54.730717 29798 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.732558 28252 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 07:59:54.900074 29443 raft_consensus.cc:482] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:54.900218 29443 raft_consensus.cc:504] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.900846 29443 leader_election.cc:288] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:54.901242 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a40382745f84f0eadf9e0ee2e057289" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:54.901252 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a40382745f84f0eadf9e0ee2e057289" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:54.901388 28530 raft_consensus.cc:2454] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:54.901427 28397 raft_consensus.cc:2454] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 0.
I20240502 07:59:54.901717 28725 leader_election.cc:302] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:54.901916 29443 raft_consensus.cc:2781] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:54.901975 29443 raft_consensus.cc:482] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:54.902030 29443 raft_consensus.cc:3037] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.902856 29443 raft_consensus.cc:504] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.903177 29443 leader_election.cc:288] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:54.903576 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a40382745f84f0eadf9e0ee2e057289" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:54.903581 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a40382745f84f0eadf9e0ee2e057289" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:54.903697 28530 raft_consensus.cc:3037] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.903708 28397 raft_consensus.cc:3037] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.904527 28530 raft_consensus.cc:2454] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:54.904806 28725 leader_election.cc:302] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:54.904963 29443 raft_consensus.cc:2781] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:54.904953 28397 raft_consensus.cc:2454] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 1.
I20240502 07:59:54.905165 29443 raft_consensus.cc:686] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 07:59:54.905333 29443 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:54.907521 28252 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 07:59:54.988233 29803 raft_consensus.cc:482] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:54.988380 29803 raft_consensus.cc:504] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.988869 29803 leader_election.cc:288] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.989346 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698f494a22d34e659766b97cb24d0293" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:54.989346 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698f494a22d34e659766b97cb24d0293" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:54.989486 28791 raft_consensus.cc:2454] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:54.989486 28397 raft_consensus.cc:2454] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:54.989804 28595 leader_election.cc:302] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:54.990016 29803 raft_consensus.cc:2781] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:54.990077 29803 raft_consensus.cc:482] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:54.990130 29803 raft_consensus.cc:3037] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.990936 29803 raft_consensus.cc:504] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.991295 29803 leader_election.cc:288] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:54.991675 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698f494a22d34e659766b97cb24d0293" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:54.991701 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "698f494a22d34e659766b97cb24d0293" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:54.991802 28397 raft_consensus.cc:3037] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.991789 28791 raft_consensus.cc:3037] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:54.992763 28397 raft_consensus.cc:2454] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:54.992763 28791 raft_consensus.cc:2454] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:54.993095 28597 leader_election.cc:302] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:54.993324 29803 raft_consensus.cc:2781] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:54.993451 29803 raft_consensus.cc:686] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:54.993605 29803 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:54.995409 28252 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 07:59:55.187173 29832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:55.187484 29832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:55.191196 29832 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240502 07:59:55.217741 29628 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.221175 29628 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.294095 28530 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.294571 29443 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.295399 28397 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.296856 29443 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.297621 28792 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.300559 29803 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.303323 28397 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.305372 29692 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.315984 29874 mvcc.cc:205] Tried to move back new op lower bound from 7023152313549172736 to 7023152312294420480. Current Snapshot: MvccSnapshot[applied={T|T < 7023152313533747200}]
I20240502 07:59:55.316244 29873 mvcc.cc:205] Tried to move back new op lower bound from 7023152313549172736 to 7023152312294420480. Current Snapshot: MvccSnapshot[applied={T|T < 7023152313533747200}]
I20240502 07:59:55.322686 29856 mvcc.cc:205] Tried to move back new op lower bound from 7023152313549172736 to 7023152312294420480. Current Snapshot: MvccSnapshot[applied={T|T < 7023152313533747200}]
INSERT report
rows total: 2000
time total: 117.268 ms
time per row: 0.058634 ms
I20240502 07:59:55.356057 28256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:55.362406 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb29b8b566747e5b82a8969a070123e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.362771 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29b8b566747e5b82a8969a070123e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.362763 28640 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb29b8b566747e5b82a8969a070123e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.363024 28639 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1cd0ec72d1429f9621d217b1bf845a (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.363340 28641 tablet_service.cc:1467] Processing CreateTablet for tablet 57f7c358ace742649df52380939ece9d (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.363420 28771 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1cd0ec72d1429f9621d217b1bf845a (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.363498 28772 tablet_service.cc:1467] Processing CreateTablet for tablet 57f7c358ace742649df52380939ece9d (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.363677 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1cd0ec72d1429f9621d217b1bf845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.363101 28640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29b8b566747e5b82a8969a070123e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.363842 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f7c358ace742649df52380939ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.364559 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f7c358ace742649df52380939ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.365934 29894 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:55.366016 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1cd0ec72d1429f9621d217b1bf845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.366530 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 9e1cd0ec72d1429f9621d217b1bf845a (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.366763 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1cd0ec72d1429f9621d217b1bf845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.366827 29894 tablet_bootstrap.cc:654] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.368345 29893 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:55.368806 29894 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:55.368983 29894 ts_tablet_manager.cc:1367] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 07:59:55.369491 29893 tablet_bootstrap.cc:654] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.369467 28377 tablet_service.cc:1467] Processing CreateTablet for tablet 4cb29b8b566747e5b82a8969a070123e (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.369493 29894 raft_consensus.cc:348] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.369599 29894 raft_consensus.cc:374] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.369629 29894 raft_consensus.cc:729] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.369767 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29b8b566747e5b82a8969a070123e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.369788 29894 consensus_queue.cc:260] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.369467 28378 tablet_service.cc:1467] Processing CreateTablet for tablet 57f7c358ace742649df52380939ece9d (DEFAULT_TABLE table=rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.370019 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f7c358ace742649df52380939ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.371344 29894 ts_tablet_manager.cc:1398] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:55.371666 29894 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:55.372112 29895 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:55.372321 29898 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:55.373140 29895 tablet_bootstrap.cc:654] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.374096 29894 tablet_bootstrap.cc:654] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.374577 29895 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:55.374677 29895 ts_tablet_manager.cc:1367] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:55.375190 29895 raft_consensus.cc:348] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.375293 29895 raft_consensus.cc:374] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.375305 29893 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:55.375329 29895 raft_consensus.cc:729] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.375391 29893 ts_tablet_manager.cc:1367] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 07:59:55.375458 29895 consensus_queue.cc:260] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.375686 29894 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:55.375752 29894 ts_tablet_manager.cc:1367] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 07:59:55.375762 29895 ts_tablet_manager.cc:1398] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.375842 29893 raft_consensus.cc:348] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.375934 29895 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:55.375946 29893 raft_consensus.cc:374] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.375982 29893 raft_consensus.cc:729] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.376106 29893 consensus_queue.cc:260] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.376148 29894 raft_consensus.cc:348] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.376240 29894 raft_consensus.cc:374] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.376266 29894 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.376365 29894 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.376612 29894 ts_tablet_manager.cc:1398] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.376895 29895 tablet_bootstrap.cc:654] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.376986 29893 ts_tablet_manager.cc:1398] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:55.377151 29893 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:55.378114 29893 tablet_bootstrap.cc:654] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.378976 29895 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:55.379055 29895 ts_tablet_manager.cc:1367] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:55.379462 29895 raft_consensus.cc:348] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.379573 29895 raft_consensus.cc:374] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.379606 29895 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.379716 29895 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.380038 29895 ts_tablet_manager.cc:1398] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:55.380200 29895 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:55.381011 29895 tablet_bootstrap.cc:654] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.382316 29893 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:55.382464 29893 ts_tablet_manager.cc:1367] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:55.382995 29893 raft_consensus.cc:348] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.383180 29893 raft_consensus.cc:374] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.383276 29893 raft_consensus.cc:729] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.383316 29895 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:55.383402 29895 ts_tablet_manager.cc:1367] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:55.383481 29893 consensus_queue.cc:260] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.383879 29895 raft_consensus.cc:348] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.383978 29895 raft_consensus.cc:374] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.384011 29895 raft_consensus.cc:729] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.384114 29895 consensus_queue.cc:260] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.384418 29895 ts_tablet_manager.cc:1398] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.385556 29893 ts_tablet_manager.cc:1398] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:55.388950 29898 tablet_bootstrap.cc:654] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.390146 29898 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:55.390240 29898 ts_tablet_manager.cc:1367] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.000s
I20240502 07:59:55.390687 29898 raft_consensus.cc:348] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.390805 29898 raft_consensus.cc:374] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.390849 29898 raft_consensus.cc:729] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.390982 29898 consensus_queue.cc:260] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.391250 29898 ts_tablet_manager.cc:1398] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.391409 29898 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:55.392300 29898 tablet_bootstrap.cc:654] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.393895 29898 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:55.393980 29898 ts_tablet_manager.cc:1367] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:55.394407 29898 raft_consensus.cc:348] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.394532 29898 raft_consensus.cc:374] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.394565 29898 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.394711 29898 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.394977 29898 ts_tablet_manager.cc:1398] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:55.494180 29803 raft_consensus.cc:482] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:55.494325 29803 raft_consensus.cc:504] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.494778 29803 leader_election.cc:288] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:55.495179 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:55.495245 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:55.495337 28791 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:55.495370 28398 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:55.495643 28597 leader_election.cc:302] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:55.495827 29803 raft_consensus.cc:2781] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:55.495893 29803 raft_consensus.cc:482] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:55.495929 29803 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.496822 29803 raft_consensus.cc:504] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.497161 29803 leader_election.cc:288] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:55.497507 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:55.497509 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:55.497615 28398 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.497651 28791 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.498440 28791 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:55.498453 28398 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:55.498764 28595 leader_election.cc:302] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:55.498966 29803 raft_consensus.cc:2781] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:55.499078 29803 raft_consensus.cc:686] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:55.499239 29803 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 07:59:55.501122 28255 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 07:59:55.517449 29803 raft_consensus.cc:482] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:55.517567 29803 raft_consensus.cc:504] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.517982 29803 leader_election.cc:288] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:55.518335 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:55.518324 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:55.518472 28398 raft_consensus.cc:2454] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:55.518473 28530 raft_consensus.cc:2454] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:55.518733 28595 leader_election.cc:302] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:55.518908 29803 raft_consensus.cc:2781] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:55.518996 29803 raft_consensus.cc:482] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:55.519024 29803 raft_consensus.cc:3037] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.519869 29803 raft_consensus.cc:504] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.520216 29803 leader_election.cc:288] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:55.520592 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:55.520738 28530 raft_consensus.cc:3037] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.520870 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:55.521021 28398 raft_consensus.cc:3037] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.521847 28530 raft_consensus.cc:2454] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:55.521941 28398 raft_consensus.cc:2454] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:55.522161 28594 leader_election.cc:302] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:55.522333 29803 raft_consensus.cc:2781] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:55.522434 29803 raft_consensus.cc:686] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:55.522579 29803 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.524355 28255 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 07:59:55.641345 29628 raft_consensus.cc:482] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:55.641499 29628 raft_consensus.cc:504] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.642139 29628 leader_election.cc:288] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:55.642598 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1cd0ec72d1429f9621d217b1bf845a" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:55.642601 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1cd0ec72d1429f9621d217b1bf845a" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:55.642746 28791 raft_consensus.cc:2454] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:55.642781 28660 raft_consensus.cc:2454] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:55.643100 28466 leader_election.cc:302] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:55.643314 29628 raft_consensus.cc:2781] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:55.643383 29628 raft_consensus.cc:482] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:55.643409 29628 raft_consensus.cc:3037] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.644181 29628 raft_consensus.cc:504] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.644567 29628 leader_election.cc:288] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:55.644860 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1cd0ec72d1429f9621d217b1bf845a" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:55.644951 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e1cd0ec72d1429f9621d217b1bf845a" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:55.644991 28660 raft_consensus.cc:3037] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.645077 28791 raft_consensus.cc:3037] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:55.646046 28660 raft_consensus.cc:2454] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:55.646046 28791 raft_consensus.cc:2454] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:55.646378 28465 leader_election.cc:302] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:55.646570 29628 raft_consensus.cc:2781] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:55.646667 29628 raft_consensus.cc:686] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:55.646840 29628 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:55.648593 28255 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 07:59:55.781265 29909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:55.781564 29909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:55.784932 29909 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240502 07:59:55.885152 28530 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.885152 28398 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.885648 29692 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.885803 29803 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.888619 28791 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.889294 28660 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.889966 29628 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.890659 29628 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.893807 28792 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:55.894240 29803 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:55.894438 29692 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 122.807 ms
time per row: 0.0614033 ms
I20240502 07:59:55.955009 28250 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:55.960711 28377 tablet_service.cc:1467] Processing CreateTablet for tablet a28d406b43194bc6b775707d021618f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.960711 28772 tablet_service.cc:1467] Processing CreateTablet for tablet a28d406b43194bc6b775707d021618f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.961035 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28d406b43194bc6b775707d021618f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.961035 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28d406b43194bc6b775707d021618f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.960958 28509 tablet_service.cc:1467] Processing CreateTablet for tablet a28d406b43194bc6b775707d021618f0 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.961257 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28d406b43194bc6b775707d021618f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.961974 28508 tablet_service.cc:1467] Processing CreateTablet for tablet 07461c988fb3402c91cfd9235efbb9b1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.962189 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07461c988fb3402c91cfd9235efbb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.963373 29971 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:55.963624 29972 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:55.963666 28377 tablet_service.cc:1467] Processing CreateTablet for tablet 07461c988fb3402c91cfd9235efbb9b1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.963900 28377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07461c988fb3402c91cfd9235efbb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.964352 29971 tablet_bootstrap.cc:654] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.964567 29972 tablet_bootstrap.cc:654] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.964537 28378 tablet_service.cc:1467] Processing CreateTablet for tablet dcd68a21ed7b4205a2f0cf75f80df0e6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.964766 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68a21ed7b4205a2f0cf75f80df0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.965667 29972 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:55.965763 29972 ts_tablet_manager.cc:1367] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:55.966228 29971 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:55.966203 29972 raft_consensus.cc:348] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.966296 29971 ts_tablet_manager.cc:1367] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:55.966316 29972 raft_consensus.cc:374] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.966356 29972 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.966483 29972 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.966701 29971 raft_consensus.cc:348] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.966794 29972 ts_tablet_manager.cc:1398] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:55.966809 29971 raft_consensus.cc:374] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.966841 29971 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.966960 29972 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:55.966974 29971 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.967219 29971 ts_tablet_manager.cc:1398] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:55.967383 29971 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:55.967916 29972 tablet_bootstrap.cc:654] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.968290 29971 tablet_bootstrap.cc:654] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.968982 29972 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:55.969056 29972 ts_tablet_manager.cc:1367] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 07:59:55.969286 28772 tablet_service.cc:1467] Processing CreateTablet for tablet dcd68a21ed7b4205a2f0cf75f80df0e6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.969548 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68a21ed7b4205a2f0cf75f80df0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.969502 29972 raft_consensus.cc:348] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.969602 29972 raft_consensus.cc:374] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.969635 29972 raft_consensus.cc:729] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.969756 29972 consensus_queue.cc:260] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.970033 29972 ts_tablet_manager.cc:1398] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:55.969971 28639 tablet_service.cc:1467] Processing CreateTablet for tablet 07461c988fb3402c91cfd9235efbb9b1 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.970243 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07461c988fb3402c91cfd9235efbb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.970613 29971 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:55.970686 29971 ts_tablet_manager.cc:1367] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 07:59:55.971081 29971 raft_consensus.cc:348] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.971174 29971 raft_consensus.cc:374] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.971202 29971 raft_consensus.cc:729] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.971324 29971 consensus_queue.cc:260] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.971607 29971 ts_tablet_manager.cc:1398] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:55.971769 29971 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:55.972594 29978 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:55.972671 28641 tablet_service.cc:1467] Processing CreateTablet for tablet dcd68a21ed7b4205a2f0cf75f80df0e6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:55.972759 29971 tablet_bootstrap.cc:654] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.972906 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68a21ed7b4205a2f0cf75f80df0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:55.973529 29978 tablet_bootstrap.cc:654] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.974539 29978 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:55.974627 29978 ts_tablet_manager.cc:1367] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:55.975028 29978 raft_consensus.cc:348] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.975139 29978 raft_consensus.cc:374] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.975183 29978 raft_consensus.cc:729] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.975323 29978 consensus_queue.cc:260] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.975697 29978 ts_tablet_manager.cc:1398] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.975787 29970 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:55.975859 29978 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:55.976956 29978 tablet_bootstrap.cc:654] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.978401 29978 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:55.978502 29978 ts_tablet_manager.cc:1367] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:55.978782 29970 tablet_bootstrap.cc:654] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.979004 29978 raft_consensus.cc:348] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.979128 29978 raft_consensus.cc:374] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.979179 29978 raft_consensus.cc:729] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.979307 29978 consensus_queue.cc:260] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.979648 29978 ts_tablet_manager.cc:1398] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.980036 29971 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:55.980141 29971 ts_tablet_manager.cc:1367] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 07:59:55.980571 29970 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:55.980703 29970 ts_tablet_manager.cc:1367] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 07:59:55.980880 29971 raft_consensus.cc:348] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.980983 29971 raft_consensus.cc:374] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.981015 29971 raft_consensus.cc:729] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.981142 29971 consensus_queue.cc:260] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.981426 29971 ts_tablet_manager.cc:1398] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:55.981196 29970 raft_consensus.cc:348] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.981549 29970 raft_consensus.cc:374] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.981595 29970 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.981918 29970 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:55.982208 29970 ts_tablet_manager.cc:1398] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:55.982344 29970 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:55.983186 29970 tablet_bootstrap.cc:654] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:55.984239 29970 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:55.984328 29970 ts_tablet_manager.cc:1367] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:55.984797 29970 raft_consensus.cc:348] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.984892 29970 raft_consensus.cc:374] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:55.988564 29970 raft_consensus.cc:729] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:55.988708 29970 consensus_queue.cc:260] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:55.988992 29970 ts_tablet_manager.cc:1398] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 07:59:56.081413 29975 raft_consensus.cc:482] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.081552 29975 raft_consensus.cc:504] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.081956 29975 leader_election.cc:288] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:56.082422 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28d406b43194bc6b775707d021618f0" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:56.082417 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28d406b43194bc6b775707d021618f0" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:56.082558 28530 raft_consensus.cc:2454] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:56.082569 28792 raft_consensus.cc:2454] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:56.082909 28335 leader_election.cc:302] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:56.083097 29975 raft_consensus.cc:2781] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.083155 29975 raft_consensus.cc:482] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.083199 29975 raft_consensus.cc:3037] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.084102 29975 raft_consensus.cc:504] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.084439 29975 leader_election.cc:288] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:56.084800 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28d406b43194bc6b775707d021618f0" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:56.084892 28792 raft_consensus.cc:3037] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.084854 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28d406b43194bc6b775707d021618f0" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:56.084985 28530 raft_consensus.cc:3037] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.085812 28792 raft_consensus.cc:2454] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:56.086017 28530 raft_consensus.cc:2454] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:56.086138 28335 leader_election.cc:302] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:56.086295 29643 raft_consensus.cc:2781] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:56.086372 29643 raft_consensus.cc:686] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 07:59:56.086506 29643 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.089217 28254 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 07:59:56.098973 29964 raft_consensus.cc:482] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.099092 29964 raft_consensus.cc:504] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.099498 29964 leader_election.cc:288] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:56.099858 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68a21ed7b4205a2f0cf75f80df0e6" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:56.099860 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68a21ed7b4205a2f0cf75f80df0e6" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:56.100010 28399 raft_consensus.cc:2454] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:56.100009 28792 raft_consensus.cc:2454] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:56.100353 28597 leader_election.cc:302] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:56.100538 29964 raft_consensus.cc:2781] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.100598 29964 raft_consensus.cc:482] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.100647 29964 raft_consensus.cc:3037] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.101449 29964 raft_consensus.cc:504] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.101778 29964 leader_election.cc:288] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:56.102092 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68a21ed7b4205a2f0cf75f80df0e6" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:56.102147 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd68a21ed7b4205a2f0cf75f80df0e6" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:56.102245 28792 raft_consensus.cc:3037] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.102268 28399 raft_consensus.cc:3037] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.103135 28399 raft_consensus.cc:2454] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:56.103157 28792 raft_consensus.cc:2454] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:56.103422 28595 leader_election.cc:302] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:56.103610 29963 raft_consensus.cc:2781] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:56.103703 29963 raft_consensus.cc:686] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:56.103861 29963 consensus_queue.cc:237] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.105861 28254 catalog_manager.cc:5484] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 07:59:56.209089 29628 raft_consensus.cc:482] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.209228 29628 raft_consensus.cc:504] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.209667 29628 leader_election.cc:288] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:56.210098 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07461c988fb3402c91cfd9235efbb9b1" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:56.210176 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07461c988fb3402c91cfd9235efbb9b1" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec" is_pre_election: true
I20240502 07:59:56.210287 28660 raft_consensus.cc:2454] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:56.210333 28399 raft_consensus.cc:2454] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:56.210594 28465 leader_election.cc:302] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:56.210830 29628 raft_consensus.cc:2781] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.210896 29628 raft_consensus.cc:482] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.210937 29628 raft_consensus.cc:3037] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.211781 29628 raft_consensus.cc:504] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.212133 29628 leader_election.cc:288] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 07:59:56.212477 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07461c988fb3402c91cfd9235efbb9b1" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:56.212618 28660 raft_consensus.cc:3037] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.212638 28399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07461c988fb3402c91cfd9235efbb9b1" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 07:59:56.212769 28399 raft_consensus.cc:3037] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.213625 28660 raft_consensus.cc:2454] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:56.213917 28399 raft_consensus.cc:2454] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:56.213971 28465 leader_election.cc:302] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:56.214141 29628 raft_consensus.cc:2781] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:56.214282 29628 raft_consensus.cc:686] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:56.214473 29628 consensus_queue.cc:237] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.217332 28254 catalog_manager.cc:5484] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
W20240502 07:59:56.384912 29987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:56.385270 29987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:56.389724 29987 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 07:59:56.492228 28399 raft_consensus.cc:1261] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:56.493508 28792 raft_consensus.cc:1261] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:56.494364 29964 consensus_queue.cc:1035] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:56.494767 29964 consensus_queue.cc:1035] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:56.495290 28399 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:56.495550 28660 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:56.495658 29798 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:56.497694 29798 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:56.501459 28791 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:56.502130 29975 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:56.503862 28530 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:56.504248 29643 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 131.923 ms
time per row: 0.0659616 ms
I20240502 07:59:56.574319 28252 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57748:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 07:59:56.581976 28508 tablet_service.cc:1467] Processing CreateTablet for tablet afc9c4d9dd5b4c039686ccd918616859 (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.582098 28509 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf7921ae84476ab53fbd4707f463eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.582298 28508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc9c4d9dd5b4c039686ccd918616859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.582484 28772 tablet_service.cc:1467] Processing CreateTablet for tablet afc9c4d9dd5b4c039686ccd918616859 (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.582509 28510 tablet_service.cc:1467] Processing CreateTablet for tablet 3598dbda342f49aea2384b5beb17d7a1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.582726 28510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3598dbda342f49aea2384b5beb17d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.582743 28772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc9c4d9dd5b4c039686ccd918616859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.583178 28771 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf7921ae84476ab53fbd4707f463eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.583577 28770 tablet_service.cc:1467] Processing CreateTablet for tablet 3598dbda342f49aea2384b5beb17d7a1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.583575 28378 tablet_service.cc:1467] Processing CreateTablet for tablet 3598dbda342f49aea2384b5beb17d7a1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.583827 28378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3598dbda342f49aea2384b5beb17d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.583976 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf7921ae84476ab53fbd4707f463eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.586005 30054 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 07:59:56.587157 30054 tablet_bootstrap.cc:654] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.588065 28641 tablet_service.cc:1467] Processing CreateTablet for tablet afc9c4d9dd5b4c039686ccd918616859 (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.588284 28770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3598dbda342f49aea2384b5beb17d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.588336 28641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc9c4d9dd5b4c039686ccd918616859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.589680 30054 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: No bootstrap required, opened a new log
I20240502 07:59:56.589762 30054 ts_tablet_manager.cc:1367] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 07:59:56.590291 30054 raft_consensus.cc:348] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.590394 30054 raft_consensus.cc:374] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.590422 30054 raft_consensus.cc:729] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.590530 30054 consensus_queue.cc:260] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.590672 28639 tablet_service.cc:1467] Processing CreateTablet for tablet 7dbf7921ae84476ab53fbd4707f463eb (DEFAULT_TABLE table=rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 07:59:56.590914 28639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf7921ae84476ab53fbd4707f463eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.590920 30056 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:56.591336 30054 ts_tablet_manager.cc:1398] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:56.591737 30056 tablet_bootstrap.cc:654] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.593060 30056 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:56.593127 30056 ts_tablet_manager.cc:1367] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:56.593451 30056 raft_consensus.cc:348] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.593528 30056 raft_consensus.cc:374] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.593547 30056 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.593657 30056 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.593919 30056 ts_tablet_manager.cc:1398] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:56.594035 30056 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 07:59:56.594805 30056 tablet_bootstrap.cc:654] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.595925 30056 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: No bootstrap required, opened a new log
I20240502 07:59:56.595986 30056 ts_tablet_manager.cc:1367] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:56.596347 30056 raft_consensus.cc:348] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.596434 30056 raft_consensus.cc:374] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.596500 30056 raft_consensus.cc:729] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.596618 30056 consensus_queue.cc:260] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.596925 30056 ts_tablet_manager.cc:1398] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:56.600703 28771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf7921ae84476ab53fbd4707f463eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 07:59:56.602089 30059 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:56.602874 30059 tablet_bootstrap.cc:654] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.604491 30059 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:56.604586 30059 ts_tablet_manager.cc:1367] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:56.605044 30059 raft_consensus.cc:348] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.605157 30059 raft_consensus.cc:374] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.605181 30059 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.605276 30059 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.605520 30059 ts_tablet_manager.cc:1398] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:56.605640 30059 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:56.606505 30059 tablet_bootstrap.cc:654] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.606839 30053 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:56.608034 30059 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:56.608126 30059 ts_tablet_manager.cc:1367] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:56.608263 30053 tablet_bootstrap.cc:654] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.609335 30059 raft_consensus.cc:348] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.609447 30059 raft_consensus.cc:374] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.609519 30059 raft_consensus.cc:729] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.609642 30059 consensus_queue.cc:260] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.609972 30059 ts_tablet_manager.cc:1398] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:56.610239 30059 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 07:59:56.610707 30053 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:56.610867 30053 ts_tablet_manager.cc:1367] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 07:59:56.611158 30059 tablet_bootstrap.cc:654] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.611495 30053 raft_consensus.cc:348] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.611696 30053 raft_consensus.cc:374] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.611809 30053 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.611981 30053 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.612413 30053 ts_tablet_manager.cc:1398] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 07:59:56.612686 30053 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:56.613056 30059 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: No bootstrap required, opened a new log
I20240502 07:59:56.613131 30059 ts_tablet_manager.cc:1367] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:56.613575 30059 raft_consensus.cc:348] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.613667 30059 raft_consensus.cc:374] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.613703 30059 raft_consensus.cc:729] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.613847 30059 consensus_queue.cc:260] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.614442 30053 tablet_bootstrap.cc:654] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.614795 30059 ts_tablet_manager.cc:1398] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 07:59:56.617417 30053 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:56.617532 30053 ts_tablet_manager.cc:1367] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240502 07:59:56.618067 30053 raft_consensus.cc:348] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.618216 30053 raft_consensus.cc:374] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.618261 30053 raft_consensus.cc:729] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.618408 30053 consensus_queue.cc:260] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.618736 30053 ts_tablet_manager.cc:1398] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 07:59:56.619021 30053 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 07:59:56.620122 30053 tablet_bootstrap.cc:654] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Neither blocks nor log segments found. Creating new log.
I20240502 07:59:56.621883 30053 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: No bootstrap required, opened a new log
I20240502 07:59:56.621970 30053 ts_tablet_manager.cc:1367] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 07:59:56.622352 30053 raft_consensus.cc:348] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.622449 30053 raft_consensus.cc:374] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 07:59:56.622485 30053 raft_consensus.cc:729] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 07:59:56.622609 30053 consensus_queue.cc:260] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.622960 30053 ts_tablet_manager.cc:1398] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 07:59:56.712177 29975 raft_consensus.cc:482] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.712325 29975 raft_consensus.cc:504] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.712767 29975 leader_election.cc:288] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:56.713274 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3598dbda342f49aea2384b5beb17d7a1" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:56.713279 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3598dbda342f49aea2384b5beb17d7a1" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:56.713413 28792 raft_consensus.cc:2454] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:56.713424 28530 raft_consensus.cc:2454] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 0.
I20240502 07:59:56.713748 28335 leader_election.cc:302] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:56.713953 29975 raft_consensus.cc:2781] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.714035 29975 raft_consensus.cc:482] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.714090 29975 raft_consensus.cc:3037] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.714967 29975 raft_consensus.cc:504] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.715322 29975 leader_election.cc:288] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 07:59:56.715662 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3598dbda342f49aea2384b5beb17d7a1" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:56.715718 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3598dbda342f49aea2384b5beb17d7a1" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:56.715809 28792 raft_consensus.cc:3037] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.715827 28530 raft_consensus.cc:3037] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.716617 28792 raft_consensus.cc:2454] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:56.716638 28530 raft_consensus.cc:2454] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 1.
I20240502 07:59:56.716949 28335 leader_election.cc:302] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:56.717111 29975 raft_consensus.cc:2781] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:56.717200 29975 raft_consensus.cc:686] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 07:59:56.717358 29975 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 07:59:56.719209 28249 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 0 to 1, leader changed from <none> to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 07:59:56.744310 29798 raft_consensus.cc:482] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.744472 29798 raft_consensus.cc:504] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.744922 29798 leader_election.cc:288] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:56.745285 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf7921ae84476ab53fbd4707f463eb" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:56.745450 28792 raft_consensus.cc:2454] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:56.745496 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf7921ae84476ab53fbd4707f463eb" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:56.745646 28660 raft_consensus.cc:2454] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:56.745762 28466 leader_election.cc:302] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:56.745990 29798 raft_consensus.cc:2781] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.746043 29798 raft_consensus.cc:482] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.746070 29798 raft_consensus.cc:3037] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.746896 29798 raft_consensus.cc:504] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.747247 29798 leader_election.cc:288] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 07:59:56.747592 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf7921ae84476ab53fbd4707f463eb" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:56.747632 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7dbf7921ae84476ab53fbd4707f463eb" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:56.747707 28792 raft_consensus.cc:3037] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.747810 28660 raft_consensus.cc:3037] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.748535 28792 raft_consensus.cc:2454] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:56.748639 28660 raft_consensus.cc:2454] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 1.
I20240502 07:59:56.748909 28466 leader_election.cc:302] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 07:59:56.749143 29798 raft_consensus.cc:2781] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:56.749220 29798 raft_consensus.cc:686] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 07:59:56.749382 29798 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 07:59:56.751250 28249 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 0 to 1, leader changed from <none> to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 07:59:56.959794 29964 raft_consensus.cc:482] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.959851 29798 raft_consensus.cc:482] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 07:59:56.959975 29964 raft_consensus.cc:504] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.960000 29798 raft_consensus.cc:504] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.960505 29798 leader_election.cc:288] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:56.960876 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" is_pre_election: true
I20240502 07:59:56.960916 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:56.960950 29964 leader_election.cc:288] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:56.960990 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4" is_pre_election: true
I20240502 07:59:56.961024 28660 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 0.
I20240502 07:59:56.961035 28792 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:56.961268 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 07:59:56.961369 28465 leader_election.cc:302] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:56.961407 28530 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 0.
I20240502 07:59:56.961537 29798 raft_consensus.cc:2781] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.961535 28597 leader_election.cc:302] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 07:59:56.961709 29964 raft_consensus.cc:2781] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 07:59:56.961730 29798 raft_consensus.cc:482] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.961773 29964 raft_consensus.cc:482] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 07:59:56.961788 29798 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.961823 29964 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.962627 29964 raft_consensus.cc:504] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.962656 29798 raft_consensus.cc:504] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.963210 29798 leader_election.cc:288] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:56.963285 29964 leader_election.cc:288] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 07:59:56.963320 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 07:59:56.963362 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:56.963454 28660 raft_consensus.cc:2379] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c39bcb719db947b8afb0bb141e1f7a4e in current term 1: Already voted for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in this term.
I20240502 07:59:56.963461 28791 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 07:59:56.963762 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 07:59:56.963907 28530 raft_consensus.cc:2379] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in current term 1: Already voted for candidate c39bcb719db947b8afb0bb141e1f7a4e in this term.
I20240502 07:59:56.964341 28792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 07:59:56.964355 28791 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 1.
I20240502 07:59:56.964738 28597 leader_election.cc:302] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters: c39bcb719db947b8afb0bb141e1f7a4e
I20240502 07:59:56.964766 28466 leader_election.cc:302] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c39bcb719db947b8afb0bb141e1f7a4e; no voters: de423605477a4819a9569821f59f03e4, ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 07:59:56.964920 29964 raft_consensus.cc:2781] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 07:59:56.964990 29798 raft_consensus.cc:2726] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 07:59:56.965042 29964 raft_consensus.cc:686] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 07:59:56.965413 29964 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 07:59:56.967223 28249 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 07:59:57.160002 30070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:57.160360 30070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:57.164073 30070 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:57.179399 29975 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:57.183075 29643 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 07:59:57.263062 28660 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:57.263471 29798 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:57.263934 28790 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:57.264309 29798 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:57.267974 28792 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:57.268426 29964 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 07:59:57.269281 28530 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 07:59:57.270434 29964 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.001 ms
time per row: 0.0530003 ms
I20240502 07:59:57.327000 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:35407
--local_ip_for_outbound_sockets=127.16.36.197
--tserver_master_addrs=127.16.36.254:33923
--webserver_port=43737
--webserver_interface=127.16.36.197
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 07:59:57.447487 30133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:57.447767 30133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:57.447844 30133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:57.451143 30133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:57.451221 30133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:57.451292 30133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 07:59:57.455353 30133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:35407
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=43737
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.30133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:57.456322 30133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:57.457513 30133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:57.468133 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:57.468700 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:57.469316 30141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:57.470129 30133 server_base.cc:1034] running on GCE node
I20240502 07:59:57.470538 30133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:57.471225 30133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:57.472381 30133 hybrid_clock.cc:648] HybridClock initialized: now 1714636797472361 us; error 30 us; skew 500 ppm
I20240502 07:59:57.474524 30133 webserver.cc:460] Webserver started at http://127.16.36.197:43737/ using document root <none> and password file <none>
I20240502 07:59:57.475116 30133 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:57.475186 30133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:57.478905 30133 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20240502 07:59:57.480831 30147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.481832 30133 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 07:59:57.481945 30133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6c68c916f39a45f697df7eaf8973225a"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:57.482355 30133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:57.501528 30133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:57.502185 30133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:57.502357 30133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:57.502879 30133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:57.503777 30133 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:57.503824 30133 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.503883 30133 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:57.503916 30133 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.513058 30133 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:35407
I20240502 07:59:57.513082 30260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:35407 every 8 connection(s)
I20240502 07:59:57.514161 30133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 07:59:57.523615 30261 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:57.523753 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 30133
I20240502 07:59:57.524173 30261 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:57.524250 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.198:45421
--local_ip_for_outbound_sockets=127.16.36.198
--tserver_master_addrs=127.16.36.254:33923
--webserver_port=35347
--webserver_interface=127.16.36.198
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:57.524942 30261 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:57.526284 28249 ts_manager.cc:194] Re-registered known tserver with Master: 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 07:59:57.526938 28249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:52901
W20240502 07:59:57.645040 30264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:57.645313 30264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:57.645395 30264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:57.648803 30264 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:57.648883 30264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:57.648955 30264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.198
I20240502 07:59:57.652709 30264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.198:45421
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.16.36.198
--webserver_port=35347
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.30264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.198
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:57.653647 30264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:57.654778 30264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:57.663879 30269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:57.664538 30272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:57.665400 30270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:57.666328 30264 server_base.cc:1034] running on GCE node
I20240502 07:59:57.666806 30264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:57.667424 30264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:57.668532 30264 hybrid_clock.cc:648] HybridClock initialized: now 1714636797668505 us; error 41 us; skew 500 ppm
I20240502 07:59:57.670668 30264 webserver.cc:460] Webserver started at http://127.16.36.198:35347/ using document root <none> and password file <none>
I20240502 07:59:57.671247 30264 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:57.671330 30264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:57.675199 30264 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:57.677352 30278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.678719 30264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 07:59:57.678840 30264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6efe17dcca7b4babad4785d19552b699"
format_stamp: "Formatted at 2024-05-02 07:59:46 on dist-test-slave-9sjq"
I20240502 07:59:57.679276 30264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:57.703805 30264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:57.704510 30264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:57.704680 30264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:57.705235 30264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:57.706187 30264 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:57.706238 30264 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.706297 30264 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:57.706329 30264 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.715847 30264 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.198:45421
I20240502 07:59:57.716162 30391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.198:45421 every 8 connection(s)
I20240502 07:59:57.717072 30264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 07:59:57.721107 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 30264
I20240502 07:59:57.721524 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.199:43589
--local_ip_for_outbound_sockets=127.16.36.199
--tserver_master_addrs=127.16.36.254:33923
--webserver_port=36455
--webserver_interface=127.16.36.199
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:57.727324 30392 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:57.727591 30392 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:57.728230 30392 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:57.729401 28249 ts_manager.cc:194] Re-registered known tserver with Master: 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 07:59:57.730296 28249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.198:36769
W20240502 07:59:57.843412 30395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:57.843677 30395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:57.843725 30395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:57.847208 30395 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:57.847295 30395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:57.847388 30395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.199
I20240502 07:59:57.851753 30395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.199:43589
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.16.36.199
--webserver_port=36455
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.30395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.199
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:57.852785 30395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:57.853937 30395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:57.863178 30403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:57.863731 30401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:57.864499 30400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:57.865456 30395 server_base.cc:1034] running on GCE node
I20240502 07:59:57.865965 30395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:57.866559 30395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:57.867762 30395 hybrid_clock.cc:648] HybridClock initialized: now 1714636797867744 us; error 29 us; skew 500 ppm
I20240502 07:59:57.869910 30395 webserver.cc:460] Webserver started at http://127.16.36.199:36455/ using document root <none> and password file <none>
I20240502 07:59:57.870497 30395 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:57.870605 30395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:57.874202 30395 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20240502 07:59:57.876283 30409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.877393 30395 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 07:59:57.877512 30395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "29126d5aee924f81bdd47cc7153420a1"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:57.877948 30395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:57.889225 30395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:57.889832 30395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:57.889991 30395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:57.890517 30395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:57.891418 30395 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:57.891467 30395 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.891530 30395 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:57.891557 30395 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:57.900751 30395 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.199:43589
I20240502 07:59:57.900784 30522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.199:43589 every 8 connection(s)
I20240502 07:59:57.901808 30395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 07:59:57.908814 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 30395
I20240502 07:59:57.909246 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.200:43189
--local_ip_for_outbound_sockets=127.16.36.200
--tserver_master_addrs=127.16.36.254:33923
--webserver_port=33657
--webserver_interface=127.16.36.200
--builtin_ntp_servers=127.16.36.212:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 07:59:57.911067 30523 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:57.911371 30523 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:57.911844 30523 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:57.912974 28249 ts_manager.cc:194] Re-registered known tserver with Master: 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589)
I20240502 07:59:57.913806 28249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.199:48979
W20240502 07:59:58.031153 30526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:58.031383 30526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 07:59:58.031430 30526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:58.034543 30526 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 07:59:58.034624 30526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 07:59:58.034696 30526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.200
I20240502 07:59:58.038390 30526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.200:43189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.16.36.200
--webserver_port=33657
--tserver_master_addrs=127.16.36.254:33923
--never_fsync=true
--heap_profile_path=/tmp/kudu.30526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.200
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 07:59:58.039335 30526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 07:59:58.040524 30526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 07:59:58.053737 30534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:58.054349 30532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 07:59:58.055060 30531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 07:59:58.055729 30526 server_base.cc:1034] running on GCE node
I20240502 07:59:58.056282 30526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 07:59:58.057132 30526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 07:59:58.058324 30526 hybrid_clock.cc:648] HybridClock initialized: now 1714636798058308 us; error 40 us; skew 500 ppm
I20240502 07:59:58.061180 30526 webserver.cc:460] Webserver started at http://127.16.36.200:33657/ using document root <none> and password file <none>
I20240502 07:59:58.061954 30526 fs_manager.cc:359] Metadata directory not provided
I20240502 07:59:58.062057 30526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 07:59:58.066884 30526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 07:59:58.069658 30540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:58.071233 30526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 07:59:58.071406 30526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "e5e9553dabc94f218f6427c1685ffc38"
format_stamp: "Formatted at 2024-05-02 07:59:47 on dist-test-slave-9sjq"
I20240502 07:59:58.072057 30526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 07:59:58.100152 30526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 07:59:58.101063 30526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 07:59:58.101294 30526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 07:59:58.102095 30526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 07:59:58.103468 30526 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 07:59:58.103538 30526 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:58.103619 30526 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 07:59:58.103660 30526 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 07:59:58.116498 30526 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.200:43189
I20240502 07:59:58.116627 30653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.200:43189 every 8 connection(s)
I20240502 07:59:58.118052 30526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 07:59:58.119490 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 30526
I20240502 07:59:58.130156 30654 heartbeater.cc:347] Connected to a master server at 127.16.36.254:33923
I20240502 07:59:58.130431 30654 heartbeater.cc:464] Registering TS with master...
I20240502 07:59:58.131021 30654 heartbeater.cc:511] Master 127.16.36.254:33923 requested a full tablet report, sending...
I20240502 07:59:58.132258 28249 ts_manager.cc:194] Re-registered known tserver with Master: e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
I20240502 07:59:58.133136 28249 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.200:59679
I20240502 07:59:58.529518 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.529568 30261 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
I20240502 07:59:58.547971 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.555604 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.563604 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.571499 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:58.576970 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:58.606524 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.607439 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.676527 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.676698 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.677431 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.678083 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:58.683418 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.684516 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.694181 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.705293 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:58.733105 30392 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
I20240502 07:59:58.739938 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.752557 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.755641 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.777865 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:58.778724 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:58.780424 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.800065 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.801574 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.808493 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:58.809854 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.817843 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.821763 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.829293 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.846422 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:58.850067 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.861152 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.915928 30523 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
I20240502 07:59:58.935254 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.935957 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.939654 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.942373 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:58.944536 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:58.956650 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:58.960623 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:58.970717 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.057458 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.060631 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.077488 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.082695 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.086761 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.089828 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.092489 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.104372 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.135910 30654 heartbeater.cc:503] Master 127.16.36.254:33923 was elected leader, sending a full tablet report...
I20240502 07:59:59.221343 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.229605 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.237572 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.239791 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.245251 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.252775 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.260677 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.262053 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.277688 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.281983 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.285324 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.302119 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.305562 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.341177 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.358290 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.366750 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.377486 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.378794 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.380306 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.394253 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.401366 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.405936 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.415139 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.417279 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.417966 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.433228 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.439217 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.439716 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.450618 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.455014 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.470044 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.481254 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.630466 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.646011 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.646235 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.653066 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.658145 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.660984 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.665930 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.672536 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.704526 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.714679 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.719472 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.739904 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.742915 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.748231 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.754221 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.794261 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.824797 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.826661 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.832788 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.835541 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.856389 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.870147 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.877074 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.883955 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.950119 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.958313 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 07:59:59.967309 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.968240 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 07:59:59.980800 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 07:59:59.983085 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.988724 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.989838 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 07:59:59.995158 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.000831 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.012043 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.014683 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.032991 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.058104 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:00.062517 28530 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.068302 28661 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.070132 28398 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.070652 30129 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.070873 31416 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.071534 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.081708 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:00.083338 30129 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.085443 28661 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.092051 28255 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
W20240502 08:00:00.094755 28465 consensus_peers.cc:485] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c7e7dadff0a4cd08deb043a37a54424. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.094635 28398 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.105113 28661 consensus_queue.cc:237] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.108620 28529 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.109056 30068 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.109165 28398 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.109580 30068 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.121460 28529 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
W20240502 08:00:00.121701 28594 consensus_peers.cc:485] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d08ad2a981fc4c13bc89d22feea7d556. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.122159 31435 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.124775 28398 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.132405 28251 catalog_manager.cc:5484] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.142372 28529 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.144029 28790 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.144237 28661 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.144836 31416 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.145280 31420 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.147225 31416 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
W20240502 08:00:00.148113 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbf7921ae84476ab53fbd4707f463eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.149893 28790 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.150322 28252 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.152657 28661 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.189145 28529 consensus_queue.cc:237] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.190840 28661 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:00.191062 28465 consensus_peers.cc:485] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 167b5f3b818048af81b1350839ebe489. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.191272 31416 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.191519 28398 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.192404 31416 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.194233 30129 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.194662 28398 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.196816 28661 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.199501 28253 catalog_manager.cc:5484] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.206616 28529 consensus_queue.cc:237] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.208263 28398 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.208840 28661 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.209277 31416 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.209470 31305 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:00.214314 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07461c988fb3402c91cfd9235efbb9b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.215162 28399 consensus_queue.cc:237] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.217047 28660 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.217172 28790 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.217476 31539 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.217902 29975 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.218030 28529 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
W20240502 08:00:00.219720 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e1cd0ec72d1429f9621d217b1bf845a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.219798 28659 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.220026 28792 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.220166 31305 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.220770 31305 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.223357 31305 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.225744 28661 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
W20240502 08:00:00.226761 28335 consensus_peers.cc:485] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c47d608895f94a2c82e695ca16677abf. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.226989 28251 catalog_manager.cc:5484] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.229243 29975 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.229732 30129 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.229779 28661 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.230162 28398 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.231629 28790 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.232728 28661 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.232838 28253 catalog_manager.cc:5484] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.234337 28790 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.239920 28251 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.259050 28398 consensus_queue.cc:237] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.261881 28529 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.262863 29975 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.265465 28790 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.265985 29975 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:00.266526 28332 consensus_peers.cc:485] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1631d3e38209497ea2454fdfc555a319. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.277467 29975 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.280277 28529 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.280795 28256 catalog_manager.cc:5484] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.282737 28790 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.289249 28529 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.291350 28790 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.291823 31305 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.292021 28661 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.292542 31305 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:00.295846 28466 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0499a3f3e6bd4069bf992e32f1abfdb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.298720 28661 consensus_queue.cc:237] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.300829 28529 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.301270 28398 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.301743 30068 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.302119 31305 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.304311 28661 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.304858 28790 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.306638 31435 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.307163 28256 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
W20240502 08:00:00.311405 28596 consensus_peers.cc:485] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de90ee4ef90e4cb69ec37d7d9f0c86ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.316185 31442 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.316711 28529 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.319624 28251 catalog_manager.cc:5484] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.319626 28398 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.322757 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.331856 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.335902 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.337785 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.340859 28529 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.343132 28399 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.343274 28790 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:00.343674 28465 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d1c1fa046b741aa8cf296aa57847fe3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.343703 31420 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.344893 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:00.344923 31420 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.350021 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:00.354524 31420 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.356844 28399 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.357486 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.358085 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.358392 28249 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.360368 28790 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.415889 28399 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.419001 28660 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.419209 28661 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.419365 28529 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.419893 31551 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.420810 28790 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:00.421221 28596 consensus_peers.cc:485] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc9c4d9dd5b4c039686ccd918616859. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.421276 31551 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.421277 31442 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.421963 28529 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.422407 31442 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.424036 28661 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.425689 28399 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:00.425695 28332 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c65fe7f823df4b4996769fc784d3adb7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.425907 28528 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.426117 31435 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.426482 31435 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:00.430490 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cb29b8b566747e5b82a8969a070123e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.433514 31539 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.434165 31435 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.436398 28529 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.437093 28530 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.437399 28251 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.439533 31414 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.439651 28399 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.442113 28530 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.444739 28256 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.446295 28661 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.447444 28790 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.449561 28253 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.463380 28530 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.466337 28661 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:00.466732 28466 consensus_peers.cc:485] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04d358245cdf4b2ebb0b5005f75cf1e0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.466909 28790 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.466976 30129 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.467283 31416 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.470257 30129 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.470686 28661 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.474236 28249 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.474373 28790 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.480661 28399 consensus_queue.cc:237] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.482864 28661 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:00:00.484040 28332 consensus_peers.cc:485] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c47d608895f94a2c82e695ca16677abf. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:00.484437 28335 consensus_peers.cc:485] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c47d608895f94a2c82e695ca16677abf. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.484786 31539 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.484437 28790 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.485736 31539 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.488261 31539 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.490654 28661 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.491520 28249 catalog_manager.cc:5484] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.492225 28530 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.492715 31777 ts_tablet_manager.cc:902] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:00.493521 31777 tablet_copy_client.cc:308] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
W20240502 08:00:00.495980 28465 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d1c1fa046b741aa8cf296aa57847fe3. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:00.496264 28466 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d1c1fa046b741aa8cf296aa57847fe3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.498260 28399 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.498548 28790 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.498633 28792 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.498988 30129 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.500641 30129 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.501538 28661 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.503063 28398 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.503187 28790 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.503592 31702 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.503991 31702 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:00.504182 28594 consensus_peers.cc:485] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57f7c358ace742649df52380939ece9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.504304 28399 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.504848 28398 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.505865 28530 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.506404 29975 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:00.507378 28332 consensus_peers.cc:485] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a28d406b43194bc6b775707d021618f0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.504411 30129 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.508255 28249 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.510031 28791 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.512673 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet d08ad2a981fc4c13bc89d22feea7d556 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:56041)
I20240502 08:00:00.513267 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet d08ad2a981fc4c13bc89d22feea7d556 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:56041: session id = 29126d5aee924f81bdd47cc7153420a1-d08ad2a981fc4c13bc89d22feea7d556
I20240502 08:00:00.515220 28681 tablet_copy_source_session.cc:205] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.519397 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d08ad2a981fc4c13bc89d22feea7d556. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.519773 28792 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.520238 31775 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.527288 31777 tablet_copy_client.cc:783] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.527565 31777 tablet_copy_client.cc:647] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.530280 31777 tablet_copy_client.cc:523] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.534384 31777 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:00.534562 28398 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.536294 28792 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.537025 31702 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
W20240502 08:00:00.539027 28332 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3288b438832e488e8257227cb8934565. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.540032 31539 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.540015 28792 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.540427 31539 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.543658 28256 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.544324 28530 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.544400 31777 log.cc:824] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:00:00.544335 28398 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.544656 28529 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.544440 28790 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.548836 28249 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.550141 28398 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.552548 31775 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.556051 28792 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:00.556337 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3598dbda342f49aea2384b5beb17d7a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.556545 29975 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.557066 28529 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.557564 29975 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.560385 31551 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.561039 28253 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.561121 28529 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.565829 28792 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.574193 31849 ts_tablet_manager.cc:902] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.575039 31849 tablet_copy_client.cc:308] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.580430 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 7dbf7921ae84476ab53fbd4707f463eb from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:42745)
I20240502 08:00:00.580634 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 7dbf7921ae84476ab53fbd4707f463eb from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:42745: session id = 29126d5aee924f81bdd47cc7153420a1-7dbf7921ae84476ab53fbd4707f463eb
I20240502 08:00:00.582096 28550 tablet_copy_source_session.cc:205] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.583420 31849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf7921ae84476ab53fbd4707f463eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.585640 31849 tablet_copy_client.cc:783] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.586025 31849 tablet_copy_client.cc:647] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.588923 31849 tablet_copy_client.cc:523] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.590304 31849 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:00.593513 31539 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.594138 28790 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.596514 28530 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.597165 28256 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:00.618741 28660 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
W20240502 08:00:00.620978 28596 consensus_peers.cc:485] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc9c4d9dd5b4c039686ccd918616859. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.622162 28790 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.623435 28530 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.624567 31435 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.626034 31435 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.626462 28660 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.628082 28398 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.628618 31414 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.631709 31414 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.631995 28790 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.632537 31435 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.634698 31872 ts_tablet_manager.cc:902] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.634625 28530 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.635028 31777 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.635432 31777 tablet_bootstrap.cc:492] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:00.635829 31777 ts_tablet_manager.cc:1367] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.102s user 0.040s sys 0.001s
I20240502 08:00:00.638682 31777 raft_consensus.cc:348] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.638945 31777 raft_consensus.cc:729] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.639482 31777 consensus_queue.cc:260] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.640414 31777 ts_tablet_manager.cc:1398] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:00:00.641180 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-d08ad2a981fc4c13bc89d22feea7d556 received from {username='slave'} at 127.16.36.199:56041
I20240502 08:00:00.641290 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-d08ad2a981fc4c13bc89d22feea7d556 on tablet d08ad2a981fc4c13bc89d22feea7d556 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:00.641372 28249 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
W20240502 08:00:00.642354 28597 consensus_peers.cc:485] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afc9c4d9dd5b4c039686ccd918616859. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:00.642464 28597 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 698f494a22d34e659766b97cb24d0293. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.642843 31435 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.643640 28790 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.646128 28398 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.647490 31872 tablet_copy_client.cc:308] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.651645 28253 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.654738 28790 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.656257 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 9c7e7dadff0a4cd08deb043a37a54424 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:00:00.656376 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 9c7e7dadff0a4cd08deb043a37a54424 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-9c7e7dadff0a4cd08deb043a37a54424
I20240502 08:00:00.657752 28550 tablet_copy_source_session.cc:205] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.670380 31945 ts_tablet_manager.cc:902] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.672569 31849 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.672881 31849 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:00.673105 31849 ts_tablet_manager.cc:1367] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.083s user 0.027s sys 0.000s
I20240502 08:00:00.673779 31849 raft_consensus.cc:348] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.673902 31849 raft_consensus.cc:729] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.674059 31849 consensus_queue.cc:260] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.674918 31849 ts_tablet_manager.cc:1398] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:00:00.675942 31945 tablet_copy_client.cc:308] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.671293 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7e7dadff0a4cd08deb043a37a54424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.676139 28550 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-7dbf7921ae84476ab53fbd4707f463eb received from {username='slave'} at 127.16.36.199:42745
I20240502 08:00:00.676256 28550 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-7dbf7921ae84476ab53fbd4707f463eb on tablet 7dbf7921ae84476ab53fbd4707f463eb with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:00.688786 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 167b5f3b818048af81b1350839ebe489 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:00:00.688931 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 167b5f3b818048af81b1350839ebe489 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-167b5f3b818048af81b1350839ebe489
I20240502 08:00:00.689935 28550 tablet_copy_source_session.cc:205] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.690901 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167b5f3b818048af81b1350839ebe489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.693650 31945 tablet_copy_client.cc:783] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.693948 31872 tablet_copy_client.cc:783] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.694144 31872 tablet_copy_client.cc:647] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.694779 28660 consensus_queue.cc:237] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.697929 28398 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.699625 28530 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:00:00.700357 28596 consensus_peers.cc:485] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de90ee4ef90e4cb69ec37d7d9f0c86ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.700729 31702 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.701063 31702 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:00:00.701359 28597 consensus_peers.cc:485] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de90ee4ef90e4cb69ec37d7d9f0c86ef. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.706017 28529 consensus_queue.cc:237] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
W20240502 08:00:00.707854 28463 consensus_peers.cc:485] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7fd7f8583e14c91a932767ebe23b419. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.708504 28660 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.709290 31420 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.710355 28790 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.710835 31420 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.712162 31702 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.712663 28398 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.715020 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.715655 28530 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.716825 28252 catalog_manager.cc:5484] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:00.717289 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:00.696684 31872 tablet_copy_client.cc:523] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.724102 31872 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:00.731454 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.731751 31872 log.cc:824] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Log is configured to *not* fsync() on all Append() calls
I20240502 08:00:00.731998 31849 ts_tablet_manager.cc:902] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:00.693955 31945 tablet_copy_client.cc:647] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.732439 31849 tablet_copy_client.cc:308] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:00.741312 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 1631d3e38209497ea2454fdfc555a319 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:38205)
I20240502 08:00:00.741503 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 1631d3e38209497ea2454fdfc555a319 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:38205: session id = 29126d5aee924f81bdd47cc7153420a1-1631d3e38209497ea2454fdfc555a319
I20240502 08:00:00.743191 28419 tablet_copy_source_session.cc:205] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.744534 31945 tablet_copy_client.cc:523] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.745322 31849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1631d3e38209497ea2454fdfc555a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.745916 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.747084 31945 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:00.747717 31849 tablet_copy_client.cc:783] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.747859 31849 tablet_copy_client.cc:647] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.753028 31849 tablet_copy_client.cc:523] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.754397 31849 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:00.761185 31777 ts_tablet_manager.cc:902] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.761691 31777 tablet_copy_client.cc:308] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.762300 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 9e1cd0ec72d1429f9621d217b1bf845a from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:42745)
I20240502 08:00:00.762426 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 9e1cd0ec72d1429f9621d217b1bf845a from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:42745: session id = 29126d5aee924f81bdd47cc7153420a1-9e1cd0ec72d1429f9621d217b1bf845a
I20240502 08:00:00.762603 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.763355 28550 tablet_copy_source_session.cc:205] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.764358 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1cd0ec72d1429f9621d217b1bf845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.766567 31777 tablet_copy_client.cc:783] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.766705 31777 tablet_copy_client.cc:647] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.773556 31420 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.775753 28660 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.776770 28790 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.778400 31777 tablet_copy_client.cc:523] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.779973 31777 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:00.779867 28248 catalog_manager.cc:5484] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.805377 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:00.810693 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:00.814000 31777 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.814329 31777 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:00.814527 31777 ts_tablet_manager.cc:1367] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.000s
I20240502 08:00:00.815029 31777 raft_consensus.cc:348] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.815163 31777 raft_consensus.cc:729] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.816500 31777 consensus_queue.cc:260] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.816802 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:00.819234 31777 ts_tablet_manager.cc:1398] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 08:00:00.819574 32020 ts_tablet_manager.cc:902] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:00.822288 28550 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-9e1cd0ec72d1429f9621d217b1bf845a received from {username='slave'} at 127.16.36.199:42745
I20240502 08:00:00.822381 28550 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-9e1cd0ec72d1429f9621d217b1bf845a on tablet 9e1cd0ec72d1429f9621d217b1bf845a with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:00.824855 32020 tablet_copy_client.cc:308] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:00.839257 31872 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.839674 31872 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:00.840013 31872 ts_tablet_manager.cc:1367] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.116s user 0.033s sys 0.002s
I20240502 08:00:00.841356 32023 ts_tablet_manager.cc:902] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:00.842095 32023 tablet_copy_client.cc:308] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:00.842586 31872 raft_consensus.cc:348] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.842836 31872 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.843360 31872 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.844302 31872 ts_tablet_manager.cc:1398] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 08:00:00.845824 28550 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-9c7e7dadff0a4cd08deb043a37a54424 received from {username='slave'} at 127.16.36.198:43585
I20240502 08:00:00.845923 28550 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-9c7e7dadff0a4cd08deb043a37a54424 on tablet 9c7e7dadff0a4cd08deb043a37a54424 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:00.846901 32024 ts_tablet_manager.cc:902] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.847641 32024 tablet_copy_client.cc:308] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.850876 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet c65fe7f823df4b4996769fc784d3adb7 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:33927)
I20240502 08:00:00.850987 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet c65fe7f823df4b4996769fc784d3adb7 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:33927: session id = e5e9553dabc94f218f6427c1685ffc38-c65fe7f823df4b4996769fc784d3adb7
I20240502 08:00:00.852010 28419 tablet_copy_source_session.cc:205] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.853114 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 07461c988fb3402c91cfd9235efbb9b1 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:38023)
I20240502 08:00:00.853226 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 07461c988fb3402c91cfd9235efbb9b1 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:38023: session id = e5e9553dabc94f218f6427c1685ffc38-07461c988fb3402c91cfd9235efbb9b1
I20240502 08:00:00.854141 28550 tablet_copy_source_session.cc:205] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.856259 32023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65fe7f823df4b4996769fc784d3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.856824 32024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07461c988fb3402c91cfd9235efbb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.863258 32023 tablet_copy_client.cc:783] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.863590 32023 tablet_copy_client.cc:647] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.863659 32024 tablet_copy_client.cc:783] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.863845 32024 tablet_copy_client.cc:647] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.865789 32024 tablet_copy_client.cc:523] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.866290 31777 ts_tablet_manager.cc:902] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:00.866751 31777 tablet_copy_client.cc:308] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:00.867206 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet c47d608895f94a2c82e695ca16677abf from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:38205)
I20240502 08:00:00.867319 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet c47d608895f94a2c82e695ca16677abf from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:38205: session id = 29126d5aee924f81bdd47cc7153420a1-c47d608895f94a2c82e695ca16677abf
I20240502 08:00:00.868163 32024 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:00.868259 28419 tablet_copy_source_session.cc:205] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.868691 31945 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.869051 31945 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:00.869336 31945 ts_tablet_manager.cc:1367] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.122s user 0.031s sys 0.000s
I20240502 08:00:00.869395 31777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47d608895f94a2c82e695ca16677abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.870173 31945 raft_consensus.cc:348] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.870353 31945 raft_consensus.cc:729] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.870579 31945 consensus_queue.cc:260] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.871994 31777 tablet_copy_client.cc:783] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.872141 31777 tablet_copy_client.cc:647] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.874191 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 4cb29b8b566747e5b82a8969a070123e from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:43285)
I20240502 08:00:00.874302 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 4cb29b8b566747e5b82a8969a070123e from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:43285: session id = e5e9553dabc94f218f6427c1685ffc38-4cb29b8b566747e5b82a8969a070123e
I20240502 08:00:00.875305 28681 tablet_copy_source_session.cc:205] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.877230 32024 log.cc:824] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Log is configured to *not* fsync() on all Append() calls
I20240502 08:00:00.877660 32020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29b8b566747e5b82a8969a070123e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.879053 32023 tablet_copy_client.cc:523] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.879937 32020 tablet_copy_client.cc:783] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.880085 32020 tablet_copy_client.cc:647] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.880376 32023 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:00.882201 28660 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:00.884361 28398 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.885210 28790 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:00.886473 31435 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.886827 31435 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:00.886843 32020 tablet_copy_client.cc:523] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.888360 32020 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
W20240502 08:00:00.888340 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f98044a1730e4a04a7088199d4f0850e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.893025 31945 ts_tablet_manager.cc:1398] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20240502 08:00:00.895251 28550 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-167b5f3b818048af81b1350839ebe489 received from {username='slave'} at 127.16.36.198:43585
I20240502 08:00:00.895364 28550 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-167b5f3b818048af81b1350839ebe489 on tablet 167b5f3b818048af81b1350839ebe489 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:00.897217 31777 tablet_copy_client.cc:523] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.898180 31983 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.898690 31777 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:00.900410 32058 ts_tablet_manager.cc:902] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.900919 28790 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.903028 28398 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.905861 28252 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:00.910693 30476 raft_consensus.cc:1203] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:00.922248 32076 ts_tablet_manager.cc:902] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.923151 32076 tablet_copy_client.cc:308] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.923501 32058 tablet_copy_client.cc:308] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.928823 31945 ts_tablet_manager.cc:902] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.929369 31945 tablet_copy_client.cc:308] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.929847 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 5d1c1fa046b741aa8cf296aa57847fe3 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:00:00.929960 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 5d1c1fa046b741aa8cf296aa57847fe3 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-5d1c1fa046b741aa8cf296aa57847fe3
I20240502 08:00:00.930924 28550 tablet_copy_source_session.cc:205] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.931982 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1c1fa046b741aa8cf296aa57847fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.934324 32020 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.934450 31945 tablet_copy_client.cc:783] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.934612 31945 tablet_copy_client.cc:647] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.934926 32020 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:00.935314 32020 ts_tablet_manager.cc:1367] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.047s user 0.025s sys 0.005s
I20240502 08:00:00.939590 32020 raft_consensus.cc:348] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.939958 32020 raft_consensus.cc:729] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.941380 32020 consensus_queue.cc:260] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.942833 32020 ts_tablet_manager.cc:1398] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240502 08:00:00.944844 28398 consensus_queue.cc:237] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:00.946887 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-4cb29b8b566747e5b82a8969a070123e received from {username='slave'} at 127.16.36.200:43285
I20240502 08:00:00.946981 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-4cb29b8b566747e5b82a8969a070123e on tablet 4cb29b8b566747e5b82a8969a070123e with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:00.947221 28529 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:00:00.948205 28332 consensus_peers.cc:485] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.948591 28790 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:00.949199 31539 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.949968 31539 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:00.952509 31539 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.955152 32111 ts_tablet_manager.cc:902] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:00.955317 28248 catalog_manager.cc:5484] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.955766 32111 tablet_copy_client.cc:308] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:00.955783 28790 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.956226 31945 tablet_copy_client.cc:523] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.958184 28530 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:00.958945 31945 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:00.962206 32020 ts_tablet_manager.cc:902] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:00.962687 32020 tablet_copy_client.cc:308] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:00.963418 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 3598dbda342f49aea2384b5beb17d7a1 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:33927)
I20240502 08:00:00.963549 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 3598dbda342f49aea2384b5beb17d7a1 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:33927: session id = e5e9553dabc94f218f6427c1685ffc38-3598dbda342f49aea2384b5beb17d7a1
I20240502 08:00:00.964663 28419 tablet_copy_source_session.cc:205] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.965605 32020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3598dbda342f49aea2384b5beb17d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.968008 28550 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 5d1c1fa046b741aa8cf296aa57847fe3 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:38819)
I20240502 08:00:00.968106 28550 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 5d1c1fa046b741aa8cf296aa57847fe3 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:38819: session id = 6c68c916f39a45f697df7eaf8973225a-5d1c1fa046b741aa8cf296aa57847fe3
I20240502 08:00:00.968156 32020 tablet_copy_client.cc:783] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.968334 32020 tablet_copy_client.cc:647] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.969210 28550 tablet_copy_source_session.cc:205] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.969544 28549 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:38819)
I20240502 08:00:00.969645 28549 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 0499a3f3e6bd4069bf992e32f1abfdb0 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:38819: session id = 6c68c916f39a45f697df7eaf8973225a-0499a3f3e6bd4069bf992e32f1abfdb0
I20240502 08:00:00.970600 28549 tablet_copy_source_session.cc:205] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.970901 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:38819)
I20240502 08:00:00.971011 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 04d358245cdf4b2ebb0b5005f75cf1e0 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:38819: session id = 6c68c916f39a45f697df7eaf8973225a-04d358245cdf4b2ebb0b5005f75cf1e0
I20240502 08:00:00.971871 28548 tablet_copy_source_session.cc:205] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.978823 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0499a3f3e6bd4069bf992e32f1abfdb0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 08:00:00.981055 28334 consensus_peers.cc:485] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1631d3e38209497ea2454fdfc555a319. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:00.981359 32020 tablet_copy_client.cc:523] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:00.981822 32111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d358245cdf4b2ebb0b5005f75cf1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.982954 32020 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:00.985509 32134 ts_tablet_manager.cc:902] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:00.986001 32024 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:00.986267 32024 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:00.986461 32134 tablet_copy_client.cc:308] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:00.987005 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet a28d406b43194bc6b775707d021618f0 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:38205)
I20240502 08:00:00.987108 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet a28d406b43194bc6b775707d021618f0 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:38205: session id = 29126d5aee924f81bdd47cc7153420a1-a28d406b43194bc6b775707d021618f0
I20240502 08:00:00.988044 28419 tablet_copy_source_session.cc:205] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.989032 32076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1c1fa046b741aa8cf296aa57847fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.989077 32134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28d406b43194bc6b775707d021618f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:00.991461 32134 tablet_copy_client.cc:783] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:00.991683 32134 tablet_copy_client.cc:647] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:00.991966 32135 ts_tablet_manager.cc:902] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:00.992159 32024 ts_tablet_manager.cc:1367] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.124s user 0.034s sys 0.001s
I20240502 08:00:00.992666 32135 tablet_copy_client.cc:308] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:00.993187 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 3288b438832e488e8257227cb8934565 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:33927)
I20240502 08:00:00.993289 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 3288b438832e488e8257227cb8934565 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:33927: session id = e5e9553dabc94f218f6427c1685ffc38-3288b438832e488e8257227cb8934565
I20240502 08:00:00.993400 32024 raft_consensus.cc:348] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.993511 32024 raft_consensus.cc:729] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:00.993654 32024 consensus_queue.cc:260] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:00.994148 28419 tablet_copy_source_session.cc:205] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:00.994632 32024 ts_tablet_manager.cc:1398] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:00.995110 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-07461c988fb3402c91cfd9235efbb9b1 received from {username='slave'} at 127.16.36.200:38023
I20240502 08:00:00.995185 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-07461c988fb3402c91cfd9235efbb9b1 on tablet 07461c988fb3402c91cfd9235efbb9b1 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:00.996564 32135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3288b438832e488e8257227cb8934565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.001952 32135 tablet_copy_client.cc:783] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.003327 32134 tablet_copy_client.cc:523] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.004968 32134 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:01.009290 32135 tablet_copy_client.cc:647] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.004892 32023 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.010143 32023 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:01.010417 32023 ts_tablet_manager.cc:1367] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.130s user 0.022s sys 0.005s
I20240502 08:00:01.010994 32023 raft_consensus.cc:348] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.011212 32023 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.011443 32023 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.013437 32135 tablet_copy_client.cc:523] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.013881 32023 ts_tablet_manager.cc:1398] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:00:01.015126 32135 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:01.015892 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-c65fe7f823df4b4996769fc784d3adb7 received from {username='slave'} at 127.16.36.200:33927
I20240502 08:00:01.015997 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-c65fe7f823df4b4996769fc784d3adb7 on tablet c65fe7f823df4b4996769fc784d3adb7 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:01.022023 32111 tablet_copy_client.cc:783] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.022352 32111 tablet_copy_client.cc:647] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.023113 32076 tablet_copy_client.cc:783] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.023330 32076 tablet_copy_client.cc:647] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.023741 32058 tablet_copy_client.cc:783] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.023916 32058 tablet_copy_client.cc:647] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.024629 32155 ts_tablet_manager.cc:902] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:01.025285 32155 tablet_copy_client.cc:308] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:01.027460 32058 tablet_copy_client.cc:523] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.029747 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.029904 32058 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.035343 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.036561 32111 tablet_copy_client.cc:523] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.038393 32111 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.040696 30346 raft_consensus.cc:1203] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.046200 32023 ts_tablet_manager.cc:902] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.046800 32023 tablet_copy_client.cc:308] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.047312 32076 tablet_copy_client.cc:523] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.047348 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 57f7c358ace742649df52380939ece9d from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:43285)
I20240502 08:00:01.047477 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 57f7c358ace742649df52380939ece9d from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:43285: session id = e5e9553dabc94f218f6427c1685ffc38-57f7c358ace742649df52380939ece9d
I20240502 08:00:01.048496 28681 tablet_copy_source_session.cc:205] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.049419 32076 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.052477 32023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f7c358ace742649df52380939ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.054383 32157 ts_tablet_manager.cc:902] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.054955 32076 log.cc:824] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:00:01.057714 32023 tablet_copy_client.cc:783] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.057963 32023 tablet_copy_client.cc:647] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.060726 30346 raft_consensus.cc:1203] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.063612 28530 consensus_queue.cc:237] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.066212 30608 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.066932 28398 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.067535 31798 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.068617 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:01.068742 31420 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.069833 28660 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.069938 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
W20240502 08:00:01.070555 28465 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07461c988fb3402c91cfd9235efbb9b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.070922 31798 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.071076 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet c47d608895f94a2c82e695ca16677abf from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:50683)
I20240502 08:00:01.071175 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet c47d608895f94a2c82e695ca16677abf from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:50683: session id = 6c68c916f39a45f697df7eaf8973225a-c47d608895f94a2c82e695ca16677abf
I20240502 08:00:01.071313 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:01.072095 28419 tablet_copy_source_session.cc:205] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.074481 31798 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.077194 30608 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.078104 28252 catalog_manager.cc:5484] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.080154 28398 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.082162 28660 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.088510 32023 tablet_copy_client.cc:523] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.091398 32023 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:01.100206 31945 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.100541 31945 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:01.100742 31945 ts_tablet_manager.cc:1367] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.142s user 0.031s sys 0.000s
I20240502 08:00:01.101894 30476 raft_consensus.cc:1203] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.104914 31945 raft_consensus.cc:348] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.106251 31945 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.109073 31945 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.113138 31945 ts_tablet_manager.cc:1398] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 08:00:01.116123 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-5d1c1fa046b741aa8cf296aa57847fe3 received from {username='slave'} at 127.16.36.198:43585
I20240502 08:00:01.116217 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-5d1c1fa046b741aa8cf296aa57847fe3 on tablet 5d1c1fa046b741aa8cf296aa57847fe3 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:01.054998 32157 tablet_copy_client.cc:308] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.061388 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.073063 32155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c47d608895f94a2c82e695ca16677abf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.128945 32155 tablet_copy_client.cc:783] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.129141 32155 tablet_copy_client.cc:647] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.132985 31849 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.133442 31849 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:01.134251 31849 ts_tablet_manager.cc:1367] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.380s user 0.030s sys 0.001s
I20240502 08:00:01.136152 32020 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.136613 32020 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:01.136718 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.137216 31849 raft_consensus.cc:348] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.140431 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 698f494a22d34e659766b97cb24d0293 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:40461)
I20240502 08:00:01.140578 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 698f494a22d34e659766b97cb24d0293 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:40461: session id = 6c68c916f39a45f697df7eaf8973225a-698f494a22d34e659766b97cb24d0293
I20240502 08:00:01.140702 31849 raft_consensus.cc:729] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.141146 31849 consensus_queue.cc:260] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.141502 28681 tablet_copy_source_session.cc:205] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.143015 32157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.145967 31849 ts_tablet_manager.cc:1398] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 08:00:01.146538 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-1631d3e38209497ea2454fdfc555a319 received from {username='slave'} at 127.16.36.199:38205
I20240502 08:00:01.146653 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-1631d3e38209497ea2454fdfc555a319 on tablet 1631d3e38209497ea2454fdfc555a319 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:01.148792 31849 ts_tablet_manager.cc:902] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:01.149137 32155 tablet_copy_client.cc:523] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.149528 32157 tablet_copy_client.cc:783] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.149677 32157 tablet_copy_client.cc:647] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.150524 31849 tablet_copy_client.cc:308] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:01.150607 32155 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.150880 32157 tablet_copy_client.cc:523] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.151057 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet a7fd7f8583e14c91a932767ebe23b419 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:42745)
I20240502 08:00:01.151155 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet a7fd7f8583e14c91a932767ebe23b419 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:42745: session id = 29126d5aee924f81bdd47cc7153420a1-a7fd7f8583e14c91a932767ebe23b419
I20240502 08:00:01.152225 28548 tablet_copy_source_session.cc:205] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.155158 31777 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.155566 31777 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:01.155858 31777 ts_tablet_manager.cc:1367] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.257s user 0.030s sys 0.000s
I20240502 08:00:01.156370 31777 raft_consensus.cc:348] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.159559 31849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fd7f8583e14c91a932767ebe23b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.166296 32157 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.166666 31777 raft_consensus.cc:729] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.166839 31849 tablet_copy_client.cc:783] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.167165 31849 tablet_copy_client.cc:647] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.167337 31777 consensus_queue.cc:260] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.163060 28790 consensus_queue.cc:237] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.171744 31849 tablet_copy_client.cc:523] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.172915 32135 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.173395 32135 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:01.173900 31777 ts_tablet_manager.cc:1398] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.018s user 0.001s sys 0.000s
I20240502 08:00:01.174471 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-c47d608895f94a2c82e695ca16677abf received from {username='slave'} at 127.16.36.199:38205
I20240502 08:00:01.174557 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-c47d608895f94a2c82e695ca16677abf on tablet c47d608895f94a2c82e695ca16677abf with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:01.174566 31945 ts_tablet_manager.cc:902] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.175505 28660 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.176093 31945 tablet_copy_client.cc:308] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.176157 28530 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.176407 30071 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.176656 32135 ts_tablet_manager.cc:1367] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.162s user 0.026s sys 0.002s
I20240502 08:00:01.177196 32135 raft_consensus.cc:348] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.177407 30071 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.178025 31849 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:01.180023 32134 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.180392 32134 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:01.180644 32134 ts_tablet_manager.cc:1367] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.176s user 0.024s sys 0.006s
I20240502 08:00:01.181294 32134 raft_consensus.cc:348] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.181365 32219 ts_tablet_manager.cc:902] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.181499 32134 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.181711 32134 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.182128 32219 tablet_copy_client.cc:308] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.182510 32135 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.182663 32134 ts_tablet_manager.cc:1398] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:01.182749 32135 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.183274 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-a28d406b43194bc6b775707d021618f0 received from {username='slave'} at 127.16.36.199:38205
I20240502 08:00:01.183364 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-a28d406b43194bc6b775707d021618f0 on tablet a28d406b43194bc6b775707d021618f0 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:01.183647 32135 ts_tablet_manager.cc:1398] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 08:00:01.184142 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-3288b438832e488e8257227cb8934565 received from {username='slave'} at 127.16.36.200:33927
I20240502 08:00:01.184207 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-3288b438832e488e8257227cb8934565 on tablet 3288b438832e488e8257227cb8934565 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:01.184746 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet afc9c4d9dd5b4c039686ccd918616859 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:40461)
I20240502 08:00:01.184866 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet afc9c4d9dd5b4c039686ccd918616859 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:40461: session id = 6c68c916f39a45f697df7eaf8973225a-afc9c4d9dd5b4c039686ccd918616859
I20240502 08:00:01.185827 28681 tablet_copy_source_session.cc:205] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.186743 32219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc9c4d9dd5b4c039686ccd918616859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.189169 32219 tablet_copy_client.cc:783] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.189402 32219 tablet_copy_client.cc:647] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.193924 32219 tablet_copy_client.cc:523] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.195580 32219 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.195668 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet afc9c4d9dd5b4c039686ccd918616859 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:47099)
I20240502 08:00:01.195782 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet afc9c4d9dd5b4c039686ccd918616859 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:47099: session id = 6efe17dcca7b4babad4785d19552b699-afc9c4d9dd5b4c039686ccd918616859
I20240502 08:00:01.196862 28681 tablet_copy_source_session.cc:205] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.197953 31945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc9c4d9dd5b4c039686ccd918616859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.200486 31945 tablet_copy_client.cc:783] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.200773 31945 tablet_copy_client.cc:647] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.201956 31945 tablet_copy_client.cc:523] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.203492 31945 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:01.205055 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.207870 32222 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.213480 28530 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.216135 28790 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:00:01.216394 28463 consensus_peers.cc:485] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04d358245cdf4b2ebb0b5005f75cf1e0. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:01.216522 28466 consensus_peers.cc:485] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.216791 31420 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.217252 28660 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.217818 31420 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.218479 28530 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.219460 31420 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.221307 28660 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.221766 28661 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.224890 28253 catalog_manager.cc:5484] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.228878 28790 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.232254 28255 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.239796 32020 ts_tablet_manager.cc:1367] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.257s user 0.022s sys 0.003s
I20240502 08:00:01.240491 32020 raft_consensus.cc:348] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.240782 32020 raft_consensus.cc:729] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.240971 32020 consensus_queue.cc:260] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.243239 32020 ts_tablet_manager.cc:1398] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:00:01.243856 31849 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.244326 31849 tablet_bootstrap.cc:492] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:01.244307 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-3598dbda342f49aea2384b5beb17d7a1 received from {username='slave'} at 127.16.36.200:33927
I20240502 08:00:01.244396 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-3598dbda342f49aea2384b5beb17d7a1 on tablet 3598dbda342f49aea2384b5beb17d7a1 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:01.244663 31849 ts_tablet_manager.cc:1367] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.067s user 0.028s sys 0.003s
I20240502 08:00:01.245183 31849 raft_consensus.cc:348] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.245395 31849 raft_consensus.cc:729] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.245610 31849 consensus_queue.cc:260] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.246048 31849 ts_tablet_manager.cc:1398] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.249065 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-a7fd7f8583e14c91a932767ebe23b419 received from {username='slave'} at 127.16.36.199:42745
I20240502 08:00:01.249171 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-a7fd7f8583e14c91a932767ebe23b419 on tablet a7fd7f8583e14c91a932767ebe23b419 with peer 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.249827 28728 consensus_peers.cc:485] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14bbfb3758e04aeab0df4bbab8380061. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.254673 32219 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.255157 32219 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.255545 32219 ts_tablet_manager.cc:1367] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.060s user 0.023s sys 0.008s
I20240502 08:00:01.258491 32219 raft_consensus.cc:348] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.258773 31872 ts_tablet_manager.cc:902] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.258869 32219 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.259370 31872 tablet_copy_client.cc:308] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.259474 32219 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.259923 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:47099)
I20240502 08:00:01.260033 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet de90ee4ef90e4cb69ec37d7d9f0c86ef from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:47099: session id = 6efe17dcca7b4babad4785d19552b699-de90ee4ef90e4cb69ec37d7d9f0c86ef
I20240502 08:00:01.260605 32219 ts_tablet_manager.cc:1398] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240502 08:00:01.261013 28681 tablet_copy_source_session.cc:205] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.263254 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.266136 31872 tablet_copy_client.cc:783] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.266355 31872 tablet_copy_client.cc:647] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.266732 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-afc9c4d9dd5b4c039686ccd918616859 received from {username='slave'} at 127.16.36.197:40461
I20240502 08:00:01.266809 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-afc9c4d9dd5b4c039686ccd918616859 on tablet afc9c4d9dd5b4c039686ccd918616859 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.268405 32219 ts_tablet_manager.cc:902] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.269007 32219 tablet_copy_client.cc:308] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.269981 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:40461)
I20240502 08:00:01.270099 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet de90ee4ef90e4cb69ec37d7d9f0c86ef from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:40461: session id = 6c68c916f39a45f697df7eaf8973225a-de90ee4ef90e4cb69ec37d7d9f0c86ef
I20240502 08:00:01.270998 28681 tablet_copy_source_session.cc:205] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.271898 32219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.274433 32219 tablet_copy_client.cc:783] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.274626 32219 tablet_copy_client.cc:647] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.275707 32219 tablet_copy_client.cc:523] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.277011 32219 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.288076 31872 tablet_copy_client.cc:523] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.289644 31872 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:01.293547 30608 raft_consensus.cc:1203] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.296142 31945 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.296512 31945 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:01.296731 31945 ts_tablet_manager.cc:1367] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.093s user 0.022s sys 0.008s
I20240502 08:00:01.297266 31945 raft_consensus.cc:348] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.297403 31945 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.297564 31945 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.298388 31945 ts_tablet_manager.cc:1398] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:01.298956 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-afc9c4d9dd5b4c039686ccd918616859 received from {username='slave'} at 127.16.36.198:47099
I20240502 08:00:01.299036 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-afc9c4d9dd5b4c039686ccd918616859 on tablet afc9c4d9dd5b4c039686ccd918616859 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:01.309700 32111 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.310029 32111 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.310201 32111 ts_tablet_manager.cc:1367] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.272s user 0.035s sys 0.000s
I20240502 08:00:01.310842 32111 raft_consensus.cc:348] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.310984 32111 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.311129 32111 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.311520 32111 ts_tablet_manager.cc:1398] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 08:00:01.317091 30608 raft_consensus.cc:1203] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.319756 32076 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.320073 32076 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.320309 32076 ts_tablet_manager.cc:1367] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.271s user 0.023s sys 0.006s
I20240502 08:00:01.320791 32076 raft_consensus.cc:348] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.320924 32076 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.321041 32076 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.321491 32076 ts_tablet_manager.cc:1398] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.327108 28530 consensus_queue.cc:237] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.328780 28398 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.329187 32181 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:01.329834 28465 consensus_peers.cc:485] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab785e07fa8248e5889bea8ac755d002. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.330185 28790 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.330554 31416 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.335000 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-5d1c1fa046b741aa8cf296aa57847fe3 received from {username='slave'} at 127.16.36.197:38819
I20240502 08:00:01.335093 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-5d1c1fa046b741aa8cf296aa57847fe3 on tablet 5d1c1fa046b741aa8cf296aa57847fe3 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.336302 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-04d358245cdf4b2ebb0b5005f75cf1e0 received from {username='slave'} at 127.16.36.197:38819
I20240502 08:00:01.336391 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-04d358245cdf4b2ebb0b5005f75cf1e0 on tablet 04d358245cdf4b2ebb0b5005f75cf1e0 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.344059 31872 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.344383 31872 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:01.345780 31416 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.348896 28790 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.351032 28398 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.354071 30476 raft_consensus.cc:1203] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.354038 28255 catalog_manager.cc:5484] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.358286 31872 ts_tablet_manager.cc:1367] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.069s user 0.030s sys 0.000s
I20240502 08:00:01.359162 31872 raft_consensus.cc:348] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.359395 31872 raft_consensus.cc:729] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.359678 31872 consensus_queue.cc:260] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.360185 31872 ts_tablet_manager.cc:1398] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:01.360745 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-de90ee4ef90e4cb69ec37d7d9f0c86ef received from {username='slave'} at 127.16.36.198:47099
I20240502 08:00:01.360827 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-de90ee4ef90e4cb69ec37d7d9f0c86ef on tablet de90ee4ef90e4cb69ec37d7d9f0c86ef with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:01.370441 28661 consensus_queue.cc:237] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.372184 28398 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.373319 28790 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.373831 31435 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:01.374533 28597 consensus_peers.cc:485] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ba62cb7ab7b429d85e0ac037c127cb4. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.375627 31435 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.379993 32204 ts_tablet_manager.cc:902] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.380445 32204 tablet_copy_client.cc:308] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.380964 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet f98044a1730e4a04a7088199d4f0850e from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:56041)
I20240502 08:00:01.381069 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet f98044a1730e4a04a7088199d4f0850e from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:56041: session id = 29126d5aee924f81bdd47cc7153420a1-f98044a1730e4a04a7088199d4f0850e
I20240502 08:00:01.381774 32155 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.382061 32155 tablet_bootstrap.cc:492] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.382071 28681 tablet_copy_source_session.cc:205] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.382242 32155 ts_tablet_manager.cc:1367] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.232s user 0.022s sys 0.005s
I20240502 08:00:01.383018 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98044a1730e4a04a7088199d4f0850e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.385288 32204 tablet_copy_client.cc:783] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.385426 32204 tablet_copy_client.cc:647] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.386624 31872 ts_tablet_manager.cc:902] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:01.387099 31872 tablet_copy_client.cc:308] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:01.391971 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 1631d3e38209497ea2454fdfc555a319 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:56805)
I20240502 08:00:01.392093 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 1631d3e38209497ea2454fdfc555a319 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:56805: session id = 6efe17dcca7b4babad4785d19552b699-1631d3e38209497ea2454fdfc555a319
I20240502 08:00:01.393026 28419 tablet_copy_source_session.cc:205] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.394104 32023 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.394153 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1631d3e38209497ea2454fdfc555a319. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.394400 32023 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:01.394606 32023 ts_tablet_manager.cc:1367] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.303s user 0.025s sys 0.000s
I20240502 08:00:01.395066 32023 raft_consensus.cc:348] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.395205 32023 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.396404 31872 tablet_copy_client.cc:783] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.396813 31872 tablet_copy_client.cc:647] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.398219 31435 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.399099 31872 tablet_copy_client.cc:523] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.400630 31872 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:01.400702 28398 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.403060 28253 catalog_manager.cc:5484] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.404227 28790 consensus_queue.cc:237] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.405265 28792 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.405421 32204 tablet_copy_client.cc:523] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.407128 32204 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:01.408313 28661 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.409217 32222 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.411036 28530 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:00:01.411502 28728 consensus_peers.cc:485] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14bbfb3758e04aeab0df4bbab8380061. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.411840 30071 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:00:01.414697 28727 consensus_peers.cc:485] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14bbfb3758e04aeab0df4bbab8380061. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.415287 30071 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.417047 30346 raft_consensus.cc:1203] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.417752 28661 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.419137 32157 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.419984 32157 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.420178 32157 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.262s user 0.025s sys 0.004s
I20240502 08:00:01.420694 32157 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.420720 28253 catalog_manager.cc:5484] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.420909 32157 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.421097 32157 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.382719 32155 raft_consensus.cc:348] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.421284 32155 raft_consensus.cc:729] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.421432 32155 consensus_queue.cc:260] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.421305 28530 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.421744 32023 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.422526 32023 ts_tablet_manager.cc:1398] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240502 08:00:01.423043 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-57f7c358ace742649df52380939ece9d received from {username='slave'} at 127.16.36.200:43285
I20240502 08:00:01.423120 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-57f7c358ace742649df52380939ece9d on tablet 57f7c358ace742649df52380939ece9d with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:01.432113 32219 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.432732 32155 ts_tablet_manager.cc:1398] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.050s user 0.003s sys 0.000s
I20240502 08:00:01.432947 32157 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 08:00:01.433987 32058 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.445576 32058 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.445576 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-698f494a22d34e659766b97cb24d0293 received from {username='slave'} at 127.16.36.197:40461
I20240502 08:00:01.445669 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-698f494a22d34e659766b97cb24d0293 on tablet 698f494a22d34e659766b97cb24d0293 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.445919 32058 ts_tablet_manager.cc:1367] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.416s user 0.027s sys 0.003s
I20240502 08:00:01.446328 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-c47d608895f94a2c82e695ca16677abf received from {username='slave'} at 127.16.36.197:50683
I20240502 08:00:01.446421 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-c47d608895f94a2c82e695ca16677abf on tablet c47d608895f94a2c82e695ca16677abf with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.446617 32058 raft_consensus.cc:348] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.446816 32058 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.447046 32058 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.450134 32058 ts_tablet_manager.cc:1398] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:00:01.450690 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-0499a3f3e6bd4069bf992e32f1abfdb0 received from {username='slave'} at 127.16.36.197:38819
I20240502 08:00:01.450776 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-0499a3f3e6bd4069bf992e32f1abfdb0 on tablet 0499a3f3e6bd4069bf992e32f1abfdb0 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.447443 32219 tablet_bootstrap.cc:492] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.451490 32219 ts_tablet_manager.cc:1367] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.175s user 0.028s sys 0.003s
I20240502 08:00:01.452004 32219 raft_consensus.cc:348] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.453514 32219 raft_consensus.cc:729] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.453745 32219 consensus_queue.cc:260] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.455524 32219 ts_tablet_manager.cc:1398] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:00:01.457300 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-de90ee4ef90e4cb69ec37d7d9f0c86ef received from {username='slave'} at 127.16.36.197:40461
I20240502 08:00:01.457393 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-de90ee4ef90e4cb69ec37d7d9f0c86ef on tablet de90ee4ef90e4cb69ec37d7d9f0c86ef with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.461030 28661 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
W20240502 08:00:01.463677 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:01.463826 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f98044a1730e4a04a7088199d4f0850e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.464362 28792 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.464391 28398 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.465099 31983 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.465371 31983 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.468212 32364 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.470674 28398 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.470865 28792 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.473435 28249 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.476263 30608 raft_consensus.cc:1203] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.484431 30476 raft_consensus.cc:1203] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.486352 31872 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.486814 31872 tablet_bootstrap.cc:492] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:01.487063 31872 ts_tablet_manager.cc:1367] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.086s user 0.029s sys 0.001s
I20240502 08:00:01.487624 31872 raft_consensus.cc:348] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.487823 31872 raft_consensus.cc:729] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.488021 31872 consensus_queue.cc:260] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.488490 31872 ts_tablet_manager.cc:1398] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.489516 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-1631d3e38209497ea2454fdfc555a319 received from {username='slave'} at 127.16.36.198:56805
I20240502 08:00:01.489614 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-1631d3e38209497ea2454fdfc555a319 on tablet 1631d3e38209497ea2454fdfc555a319 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:01.490839 30476 raft_consensus.cc:1203] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.493952 30608 raft_consensus.cc:1203] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.494307 32204 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.494766 32204 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:01.495039 32204 ts_tablet_manager.cc:1367] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.088s user 0.027s sys 0.005s
I20240502 08:00:01.495676 32204 raft_consensus.cc:348] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.495880 32204 raft_consensus.cc:729] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.496114 32204 consensus_queue.cc:260] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.496876 32204 ts_tablet_manager.cc:1398] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:01.497342 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-f98044a1730e4a04a7088199d4f0850e received from {username='slave'} at 127.16.36.199:56041
I20240502 08:00:01.497428 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-f98044a1730e4a04a7088199d4f0850e on tablet f98044a1730e4a04a7088199d4f0850e with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:01.498301 30214 raft_consensus.cc:1203] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.521016 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.524960 28792 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.525148 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.526712 28661 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.527355 32366 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.528391 28530 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.528637 30476 raft_consensus.cc:1203] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.529836 32366 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:00:01.534272 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d939e5035541a4b003c5d276493d5c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.546283 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:01.550879 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:01.559228 28792 consensus_queue.cc:237] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.562189 28398 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.562733 32222 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.562997 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.569353 28530 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.568943 30476 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:01.570016 32366 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.571226 31872 ts_tablet_manager.cc:902] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:01.571725 31872 tablet_copy_client.cc:308] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:01.572170 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 07461c988fb3402c91cfd9235efbb9b1 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:00:01.572283 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 07461c988fb3402c91cfd9235efbb9b1 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-07461c988fb3402c91cfd9235efbb9b1
W20240502 08:00:01.572300 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fddc382ab3164ccdaa86f802c83da8be. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.573295 28548 tablet_copy_source_session.cc:205] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.574064 30608 raft_consensus.cc:1203] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.575774 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07461c988fb3402c91cfd9235efbb9b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.578336 31872 tablet_copy_client.cc:783] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.578517 31872 tablet_copy_client.cc:647] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.580618 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.581388 30346 raft_consensus.cc:1203] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.588605 31872 tablet_copy_client.cc:523] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.590355 31872 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:01.594786 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.595021 32204 ts_tablet_manager.cc:902] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:01.595506 32204 tablet_copy_client.cc:308] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:01.595978 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:42745)
I20240502 08:00:01.596074 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 04d358245cdf4b2ebb0b5005f75cf1e0 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:42745: session id = 29126d5aee924f81bdd47cc7153420a1-04d358245cdf4b2ebb0b5005f75cf1e0
I20240502 08:00:01.597208 28548 tablet_copy_source_session.cc:205] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.598287 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d358245cdf4b2ebb0b5005f75cf1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.598641 32366 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.600986 32204 tablet_copy_client.cc:783] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.601176 32204 tablet_copy_client.cc:647] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.601526 28661 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.604040 28529 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.606654 28252 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.614550 30214 raft_consensus.cc:1203] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.618172 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.620447 30214 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.626086 32204 tablet_copy_client.cc:523] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.628953 32204 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:01.630134 32222 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.630806 28530 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.633725 28397 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.633910 28254 catalog_manager.cc:5484] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index -1 to 4, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.654069 28530 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:01.656344 30476 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.657265 28792 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.657895 32181 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:00:01.659152 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e1cd0ec72d1429f9621d217b1bf845a. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.659504 32181 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.661545 28661 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.661967 32181 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.665020 32181 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.667227 30476 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.668718 28254 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.670183 28792 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.674253 28661 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:01.690152 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:01.694876 31872 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.695202 31872 tablet_bootstrap.cc:492] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:01.695372 31872 ts_tablet_manager.cc:1367] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.105s user 0.027s sys 0.000s
I20240502 08:00:01.695830 31872 raft_consensus.cc:348] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.696046 31872 raft_consensus.cc:729] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.696218 31872 consensus_queue.cc:260] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.696648 31872 ts_tablet_manager.cc:1398] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:01.697479 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-07461c988fb3402c91cfd9235efbb9b1 received from {username='slave'} at 127.16.36.198:43585
I20240502 08:00:01.697556 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-07461c988fb3402c91cfd9235efbb9b1 on tablet 07461c988fb3402c91cfd9235efbb9b1 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:01.709766 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:01.709780 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.723337 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.725593 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.735716 30214 raft_consensus.cc:1203] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.741176 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.741969 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.747478 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.761304 30214 raft_consensus.cc:1203] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.765937 30214 raft_consensus.cc:1203] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.785862 32204 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.786217 32204 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:01.786446 32204 ts_tablet_manager.cc:1367] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.158s user 0.027s sys 0.000s
I20240502 08:00:01.786955 32204 raft_consensus.cc:348] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.787088 32204 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.797319 32204 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.804239 32204 ts_tablet_manager.cc:1398] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240502 08:00:01.804786 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-04d358245cdf4b2ebb0b5005f75cf1e0 received from {username='slave'} at 127.16.36.199:42745
I20240502 08:00:01.804874 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-04d358245cdf4b2ebb0b5005f75cf1e0 on tablet 04d358245cdf4b2ebb0b5005f75cf1e0 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:01.806690 28661 consensus_queue.cc:237] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:01.808801 28398 raft_consensus.cc:1261] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.809262 31983 consensus_queue.cc:1035] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.809686 28790 raft_consensus.cc:1261] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:01.810403 28597 consensus_peers.cc:485] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcd68a21ed7b4205a2f0cf75f80df0e6. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.810659 31983 consensus_queue.cc:1035] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.812593 28660 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:01.814401 28530 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.814704 30608 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.815328 31414 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.815755 28397 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:01.816509 31414 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:00:01.816711 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cb29b8b566747e5b82a8969a070123e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.816828 32385 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:01.820658 32385 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.822305 32219 ts_tablet_manager.cc:902] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:01.822788 32219 tablet_copy_client.cc:308] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:01.823619 30608 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.825340 28530 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.827760 28253 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.829955 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 14bbfb3758e04aeab0df4bbab8380061 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:55635)
I20240502 08:00:01.830135 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 14bbfb3758e04aeab0df4bbab8380061 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:55635: session id = 6c68c916f39a45f697df7eaf8973225a-14bbfb3758e04aeab0df4bbab8380061
I20240502 08:00:01.832607 28812 tablet_copy_source_session.cc:205] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.834133 32219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bbfb3758e04aeab0df4bbab8380061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.835999 28397 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:01.836400 32219 tablet_copy_client.cc:783] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.838622 32219 tablet_copy_client.cc:647] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.840179 32219 tablet_copy_client.cc:523] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.841665 32219 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.841945 32364 raft_consensus.cc:2932] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.845265 30346 raft_consensus.cc:1203] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.845165 28249 catalog_manager.cc:5484] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.845268 28398 raft_consensus.cc:2932] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.844373 28790 raft_consensus.cc:2932] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.871953 30214 raft_consensus.cc:1203] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:01.875739 28661 consensus_queue.cc:237] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
W20240502 08:00:01.877480 28597 consensus_peers.cc:485] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df20642aef7748ff8962b3095e1ab56c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:01.878938 28790 raft_consensus.cc:1261] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.879361 28398 raft_consensus.cc:1261] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:01.880052 32385 consensus_queue.cc:1035] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.880897 31414 consensus_queue.cc:1035] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:01.883105 32058 ts_tablet_manager.cc:902] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.883646 32058 tablet_copy_client.cc:308] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.884146 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:40461)
I20240502 08:00:01.884271 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 2ba62cb7ab7b429d85e0ac037c127cb4 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:40461: session id = 6c68c916f39a45f697df7eaf8973225a-2ba62cb7ab7b429d85e0ac037c127cb4
I20240502 08:00:01.885856 28681 tablet_copy_source_session.cc:205] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.887261 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba62cb7ab7b429d85e0ac037c127cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.889561 32058 tablet_copy_client.cc:783] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.889729 32058 tablet_copy_client.cc:647] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.891309 32058 tablet_copy_client.cc:523] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.892546 31872 ts_tablet_manager.cc:902] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:01.893021 31872 tablet_copy_client.cc:308] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:01.894291 32058 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:01.900300 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:01.904687 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.906608 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.914893 32385 raft_consensus.cc:2932] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.917259 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.917976 28790 raft_consensus.cc:2932] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.919507 30345 raft_consensus.cc:1203] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:01.921589 28252 catalog_manager.cc:5484] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.921939 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 14bbfb3758e04aeab0df4bbab8380061 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:47747)
I20240502 08:00:01.922032 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 14bbfb3758e04aeab0df4bbab8380061 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:47747: session id = 6efe17dcca7b4babad4785d19552b699-14bbfb3758e04aeab0df4bbab8380061
I20240502 08:00:01.922997 28812 tablet_copy_source_session.cc:205] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.924255 31872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14bbfb3758e04aeab0df4bbab8380061. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.925731 28398 raft_consensus.cc:2932] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:01.926446 31872 tablet_copy_client.cc:783] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.926589 31872 tablet_copy_client.cc:647] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.932250 31872 tablet_copy_client.cc:523] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.933754 31872 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:01.936040 32219 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:01.951182 32219 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:01.951447 32219 ts_tablet_manager.cc:1367] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.110s user 0.029s sys 0.000s
I20240502 08:00:01.952090 32219 raft_consensus.cc:348] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.952235 32219 raft_consensus.cc:729] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:01.952247 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.952414 32219 consensus_queue.cc:260] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:01.953069 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:01.952900 32219 ts_tablet_manager.cc:1398] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:01.954572 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:01.955689 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-14bbfb3758e04aeab0df4bbab8380061 received from {username='slave'} at 127.16.36.197:55635
I20240502 08:00:01.955821 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-14bbfb3758e04aeab0df4bbab8380061 on tablet 14bbfb3758e04aeab0df4bbab8380061 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.968890 32664 ts_tablet_manager.cc:902] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:01.971583 32664 tablet_copy_client.cc:308] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:01.972299 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet f98044a1730e4a04a7088199d4f0850e from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:43285)
I20240502 08:00:01.972429 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet f98044a1730e4a04a7088199d4f0850e from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:43285: session id = e5e9553dabc94f218f6427c1685ffc38-f98044a1730e4a04a7088199d4f0850e
I20240502 08:00:01.973474 28681 tablet_copy_source_session.cc:205] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:01.977385 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:01.988085 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98044a1730e4a04a7088199d4f0850e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:01.991313 32664 tablet_copy_client.cc:783] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:01.991586 32664 tablet_copy_client.cc:647] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:01.994133 32664 tablet_copy_client.cc:523] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:01.995855 32664 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:02.015425 32640 ts_tablet_manager.cc:902] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:02.016062 32640 tablet_copy_client.cc:308] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:02.016707 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet ab785e07fa8248e5889bea8ac755d002 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:00:02.016837 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet ab785e07fa8248e5889bea8ac755d002 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-ab785e07fa8248e5889bea8ac755d002
I20240502 08:00:02.017956 28548 tablet_copy_source_session.cc:205] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.018944 32640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab785e07fa8248e5889bea8ac755d002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.024897 32640 tablet_copy_client.cc:783] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.025195 32640 tablet_copy_client.cc:647] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.026423 32640 tablet_copy_client.cc:523] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.028002 32640 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:02.032871 28790 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.035454 28398 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:00:02.036206 28530 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:00:02.036365 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a40382745f84f0eadf9e0ee2e057289. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:02.036876 32223 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:02.038251 32223 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:00:02.038854 32664 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.039224 32664 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:02.039443 32664 ts_tablet_manager.cc:1367] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.044s user 0.030s sys 0.000s
I20240502 08:00:02.039907 32664 raft_consensus.cc:348] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.040092 32664 raft_consensus.cc:729] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.040287 32664 consensus_queue.cc:260] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.041447 32664 ts_tablet_manager.cc:1398] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:02.041553 32058 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.042333 32058 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:02.042435 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-f98044a1730e4a04a7088199d4f0850e received from {username='slave'} at 127.16.36.200:43285
I20240502 08:00:02.042526 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-f98044a1730e4a04a7088199d4f0850e on tablet f98044a1730e4a04a7088199d4f0850e with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:02.043195 32058 ts_tablet_manager.cc:1367] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.149s user 0.029s sys 0.000s
I20240502 08:00:02.043692 32058 raft_consensus.cc:348] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:02.043980 32058 raft_consensus.cc:729] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.044174 32058 consensus_queue.cc:260] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:02.044579 30345 raft_consensus.cc:1203] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.045095 32058 ts_tablet_manager.cc:1398] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:02.044977 30476 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:02.045554 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-2ba62cb7ab7b429d85e0ac037c127cb4 received from {username='slave'} at 127.16.36.197:40461
I20240502 08:00:02.045642 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-2ba62cb7ab7b429d85e0ac037c127cb4 on tablet 2ba62cb7ab7b429d85e0ac037c127cb4 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:02.050485 32223 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:02.051689 32204 ts_tablet_manager.cc:902] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:02.052155 32204 tablet_copy_client.cc:308] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:02.052985 28529 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:02.053395 28252 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:02.054875 28398 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:02.073104 31872 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.073417 31872 tablet_bootstrap.cc:492] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:02.073626 31872 ts_tablet_manager.cc:1367] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.140s user 0.025s sys 0.003s
I20240502 08:00:02.074213 31872 raft_consensus.cc:348] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:02.074362 31872 raft_consensus.cc:729] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.074641 31872 consensus_queue.cc:260] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:02.074981 31872 ts_tablet_manager.cc:1398] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:02.076249 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-14bbfb3758e04aeab0df4bbab8380061 received from {username='slave'} at 127.16.36.198:47747
I20240502 08:00:02.076332 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-14bbfb3758e04aeab0df4bbab8380061 on tablet 14bbfb3758e04aeab0df4bbab8380061 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:02.104671 32664 ts_tablet_manager.cc:902] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:02.105130 32664 tablet_copy_client.cc:308] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:02.117498 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet fddc382ab3164ccdaa86f802c83da8be from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:55771)
I20240502 08:00:02.117640 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet fddc382ab3164ccdaa86f802c83da8be from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:55771: session id = e5e9553dabc94f218f6427c1685ffc38-fddc382ab3164ccdaa86f802c83da8be
I20240502 08:00:02.118685 28812 tablet_copy_source_session.cc:205] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.119643 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddc382ab3164ccdaa86f802c83da8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.122334 32664 tablet_copy_client.cc:783] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.122489 32664 tablet_copy_client.cc:647] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.124294 32664 tablet_copy_client.cc:523] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.126318 32664 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:02.137845 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 21d939e5035541a4b003c5d276493d5c from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:60995)
I20240502 08:00:02.137986 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 21d939e5035541a4b003c5d276493d5c from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:60995: session id = 29126d5aee924f81bdd47cc7153420a1-21d939e5035541a4b003c5d276493d5c
I20240502 08:00:02.138926 28812 tablet_copy_source_session.cc:205] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.139927 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.142334 32204 tablet_copy_client.cc:783] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.142498 32204 tablet_copy_client.cc:647] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.144352 32204 tablet_copy_client.cc:523] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.146116 32204 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:02.145664 32640 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.149191 32640 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:02.158375 30476 raft_consensus.cc:1203] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.159891 32640 ts_tablet_manager.cc:1367] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.132s user 0.032s sys 0.000s
I20240502 08:00:02.162513 32640 raft_consensus.cc:348] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:02.162801 32640 raft_consensus.cc:729] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.164753 32640 consensus_queue.cc:260] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:02.165925 32640 ts_tablet_manager.cc:1398] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:00:02.166518 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-ab785e07fa8248e5889bea8ac755d002 received from {username='slave'} at 127.16.36.198:43585
I20240502 08:00:02.166610 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-ab785e07fa8248e5889bea8ac755d002 on tablet ab785e07fa8248e5889bea8ac755d002 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:02.180017 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.181562 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.192097 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:02.211243 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.219700 32664 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.220116 32664 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:02.220940 32664 ts_tablet_manager.cc:1367] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.095s user 0.026s sys 0.005s
I20240502 08:00:02.222343 32664 raft_consensus.cc:348] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.222572 32664 raft_consensus.cc:729] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.222816 32664 consensus_queue.cc:260] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.223206 32664 ts_tablet_manager.cc:1398] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:02.223829 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-fddc382ab3164ccdaa86f802c83da8be received from {username='slave'} at 127.16.36.200:55771
I20240502 08:00:02.223912 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-fddc382ab3164ccdaa86f802c83da8be on tablet fddc382ab3164ccdaa86f802c83da8be with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:02.234915 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.240365 32664 ts_tablet_manager.cc:902] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:02.240535 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.243696 32664 tablet_copy_client.cc:308] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:02.247732 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 9e1cd0ec72d1429f9621d217b1bf845a from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:38023)
I20240502 08:00:02.247854 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 9e1cd0ec72d1429f9621d217b1bf845a from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:38023: session id = e5e9553dabc94f218f6427c1685ffc38-9e1cd0ec72d1429f9621d217b1bf845a
I20240502 08:00:02.248872 28548 tablet_copy_source_session.cc:205] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.249815 32664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1cd0ec72d1429f9621d217b1bf845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.253230 32664 tablet_copy_client.cc:783] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.253418 32664 tablet_copy_client.cc:647] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.253829 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.255721 32664 tablet_copy_client.cc:523] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.257154 32664 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:02.259800 30214 raft_consensus.cc:1203] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.267689 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.296840 32204 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.297255 32204 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:02.297468 32204 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.151s user 0.028s sys 0.001s
I20240502 08:00:02.298350 32204 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.298579 32204 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.298866 32204 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.299219 32204 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:02.300746 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-21d939e5035541a4b003c5d276493d5c received from {username='slave'} at 127.16.36.199:60995
I20240502 08:00:02.300837 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-21d939e5035541a4b003c5d276493d5c on tablet 21d939e5035541a4b003c5d276493d5c with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:02.302784 32204 ts_tablet_manager.cc:902] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:02.303167 32204 tablet_copy_client.cc:308] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:02.303800 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 4cb29b8b566747e5b82a8969a070123e from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:56041)
I20240502 08:00:02.303906 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 4cb29b8b566747e5b82a8969a070123e from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:56041: session id = 29126d5aee924f81bdd47cc7153420a1-4cb29b8b566747e5b82a8969a070123e
I20240502 08:00:02.304883 28681 tablet_copy_source_session.cc:205] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.306241 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cb29b8b566747e5b82a8969a070123e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.310657 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.310878 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.330798 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.348167 30608 raft_consensus.cc:1203] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.359649 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.366454 32204 tablet_copy_client.cc:783] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.366662 32204 tablet_copy_client.cc:647] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.368148 32204 tablet_copy_client.cc:523] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.369726 32204 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:02.374675 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.375232 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:02.382082 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.398474 32204 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.398789 32204 tablet_bootstrap.cc:492] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:02.398962 32204 ts_tablet_manager.cc:1367] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.029s user 0.025s sys 0.000s
I20240502 08:00:02.399484 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.399446 32204 raft_consensus.cc:348] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.399514 30345 raft_consensus.cc:1203] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.399566 32204 raft_consensus.cc:729] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.399714 32204 consensus_queue.cc:260] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.399978 32204 ts_tablet_manager.cc:1398] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 08:00:02.403537 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-4cb29b8b566747e5b82a8969a070123e received from {username='slave'} at 127.16.36.199:56041
I20240502 08:00:02.403659 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-4cb29b8b566747e5b82a8969a070123e on tablet 4cb29b8b566747e5b82a8969a070123e with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:02.411090 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:02.413643 32058 ts_tablet_manager.cc:902] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:02.414155 32058 tablet_copy_client.cc:308] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:02.414695 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet dcd68a21ed7b4205a2f0cf75f80df0e6 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:40461)
I20240502 08:00:02.414813 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet dcd68a21ed7b4205a2f0cf75f80df0e6 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:40461: session id = 6c68c916f39a45f697df7eaf8973225a-dcd68a21ed7b4205a2f0cf75f80df0e6
I20240502 08:00:02.414825 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.416257 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.417271 28681 tablet_copy_source_session.cc:205] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.418017 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.418421 32058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd68a21ed7b4205a2f0cf75f80df0e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.422288 32058 tablet_copy_client.cc:783] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.422482 32058 tablet_copy_client.cc:647] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.422848 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.428156 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.431552 32058 tablet_copy_client.cc:523] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.433168 32058 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:02.434324 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.437731 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.462983 30214 raft_consensus.cc:1203] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:02.487038 32664 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.487543 32664 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:02.487798 32664 ts_tablet_manager.cc:1367] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.231s user 0.027s sys 0.001s
I20240502 08:00:02.488320 32664 raft_consensus.cc:348] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.488569 32664 raft_consensus.cc:729] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.488775 32664 consensus_queue.cc:260] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.489238 32664 ts_tablet_manager.cc:1398] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:02.489804 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-9e1cd0ec72d1429f9621d217b1bf845a received from {username='slave'} at 127.16.36.200:38023
I20240502 08:00:02.489885 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-9e1cd0ec72d1429f9621d217b1bf845a on tablet 9e1cd0ec72d1429f9621d217b1bf845a with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:02.490773 407 ts_tablet_manager.cc:902] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:02.491490 407 tablet_copy_client.cc:308] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:02.492136 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet df20642aef7748ff8962b3095e1ab56c from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:40461)
I20240502 08:00:02.492249 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet df20642aef7748ff8962b3095e1ab56c from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:40461: session id = 6c68c916f39a45f697df7eaf8973225a-df20642aef7748ff8962b3095e1ab56c
I20240502 08:00:02.493299 28681 tablet_copy_source_session.cc:205] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.494277 407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df20642aef7748ff8962b3095e1ab56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.494381 30476 raft_consensus.cc:1203] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:02.496786 407 tablet_copy_client.cc:783] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.496989 407 tablet_copy_client.cc:647] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.497395 30345 raft_consensus.cc:1203] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:02.508625 30608 raft_consensus.cc:1203] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:02.530932 32204 ts_tablet_manager.cc:902] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:02.530979 407 tablet_copy_client.cc:523] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.531395 32204 tablet_copy_client.cc:308] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:02.532487 407 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:02.532595 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 6a40382745f84f0eadf9e0ee2e057289 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:60995)
I20240502 08:00:02.532697 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 6a40382745f84f0eadf9e0ee2e057289 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:60995: session id = 29126d5aee924f81bdd47cc7153420a1-6a40382745f84f0eadf9e0ee2e057289
I20240502 08:00:02.533761 28812 tablet_copy_source_session.cc:205] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:02.535444 32204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a40382745f84f0eadf9e0ee2e057289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:02.537721 32204 tablet_copy_client.cc:783] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:02.537875 32204 tablet_copy_client.cc:647] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:02.538486 32058 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.538761 32058 tablet_bootstrap.cc:492] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:02.538928 32058 ts_tablet_manager.cc:1367] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.106s user 0.023s sys 0.002s
I20240502 08:00:02.539361 32058 raft_consensus.cc:348] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:02.539482 32058 raft_consensus.cc:729] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.539633 32058 consensus_queue.cc:260] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:02.539917 32058 ts_tablet_manager.cc:1398] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:00:02.541311 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-dcd68a21ed7b4205a2f0cf75f80df0e6 received from {username='slave'} at 127.16.36.197:40461
I20240502 08:00:02.541391 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-dcd68a21ed7b4205a2f0cf75f80df0e6 on tablet dcd68a21ed7b4205a2f0cf75f80df0e6 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:02.542562 32204 tablet_copy_client.cc:523] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:02.544443 32204 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:02.563441 407 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.563868 407 tablet_bootstrap.cc:492] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:02.564147 407 ts_tablet_manager.cc:1367] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.000s
I20240502 08:00:02.564742 407 raft_consensus.cc:348] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:02.564965 407 raft_consensus.cc:729] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.576397 32204 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:02.576781 32204 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:02.576778 407 consensus_queue.cc:260] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:02.576989 32204 ts_tablet_manager.cc:1367] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.003s
I20240502 08:00:02.578038 407 ts_tablet_manager.cc:1398] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 08:00:02.577983 32204 raft_consensus.cc:348] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.578114 32204 raft_consensus.cc:729] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:02.578277 32204 consensus_queue.cc:260] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:02.578971 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-df20642aef7748ff8962b3095e1ab56c received from {username='slave'} at 127.16.36.197:40461
I20240502 08:00:02.579069 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-df20642aef7748ff8962b3095e1ab56c on tablet df20642aef7748ff8962b3095e1ab56c with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:02.579913 32204 ts_tablet_manager.cc:1398] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:00:02.580381 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-6a40382745f84f0eadf9e0ee2e057289 received from {username='slave'} at 127.16.36.199:60995
I20240502 08:00:02.580503 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-6a40382745f84f0eadf9e0ee2e057289 on tablet 6a40382745f84f0eadf9e0ee2e057289 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:02.591538 28790 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:02.593986 30476 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:02.594852 478 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:02.595512 28398 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:02.597280 472 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:02.597872 28529 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:00:02.598029 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a40382745f84f0eadf9e0ee2e057289. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:02.598637 470 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:02.602831 472 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:02.603183 28530 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:02.606331 28253 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:02.608645 30476 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:02.609884 28398 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:02.683418 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.683418 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.697381 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.698078 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.700039 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:02.700852 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.731191 30476 raft_consensus.cc:1203] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.741456 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.776965 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.807404 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.824159 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.824378 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:02.827185 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.836299 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.838462 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.839048 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.852123 30608 raft_consensus.cc:1203] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:02.860617 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.862635 30214 raft_consensus.cc:1203] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:02.882709 30214 raft_consensus.cc:1203] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:00:02.898444 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.915982 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.920027 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.935076 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:02.941004 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:02.944932 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.956658 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.957490 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:02.961064 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:02.981263 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.000820 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.012604 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.036337 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.037429 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.043934 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.051492 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.113389 28661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57f7c358ace742649df52380939ece9d"
dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55378
I20240502 08:00:03.113610 28661 raft_consensus.cc:595] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:03.123514 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.125056 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.125315 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.135690 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.136878 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.148631 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:03.163916 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.166301 31414 raft_consensus.cc:979] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: : Instructing follower 71922d97e1a4414698ecd14a7b2dedec to start an election
I20240502 08:00:03.166407 31414 raft_consensus.cc:1067] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Signalling peer 71922d97e1a4414698ecd14a7b2dedec to start an election
I20240502 08:00:03.167194 28398 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57f7c358ace742649df52380939ece9d"
dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
from {username='slave'} at 127.16.36.195:55199
I20240502 08:00:03.167379 28398 raft_consensus.cc:482] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:03.167451 28398 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.170253 28398 raft_consensus.cc:504] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.171459 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5e9553dabc94f218f6427c1685ffc38"
I20240502 08:00:03.171650 30608 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.172931 30608 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 2.
I20240502 08:00:03.173862 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.174283 28790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 08:00:03.174432 28790 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.175662 28790 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 2.
I20240502 08:00:03.176609 28335 leader_election.cc:302] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, de423605477a4819a9569821f59f03e4, e5e9553dabc94f218f6427c1685ffc38; no voters:
I20240502 08:00:03.176908 32399 raft_consensus.cc:2781] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:03.177007 32399 raft_consensus.cc:686] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 08:00:03.177192 32399 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.180261 788 ts_tablet_manager.cc:902] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:03.184128 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f7c358ace742649df52380939ece9d" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 08:00:03.184293 28661 raft_consensus.cc:3032] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:03.184357 28661 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 08:00:03.185592 28661 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.185868 28661 raft_consensus.cc:3037] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.187081 28661 raft_consensus.cc:2454] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 2.
I20240502 08:00:03.191906 28398 leader_election.cc:288] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 2 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
I20240502 08:00:03.193413 28255 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 1 to 2, leader changed from ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:03.200628 788 tablet_copy_client.cc:308] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:03.208086 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 6a40382745f84f0eadf9e0ee2e057289 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:55771)
I20240502 08:00:03.208233 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 6a40382745f84f0eadf9e0ee2e057289 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:55771: session id = e5e9553dabc94f218f6427c1685ffc38-6a40382745f84f0eadf9e0ee2e057289
I20240502 08:00:03.209312 28812 tablet_copy_source_session.cc:205] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:03.210429 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a40382745f84f0eadf9e0ee2e057289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:03.213058 788 tablet_copy_client.cc:783] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:03.213227 788 tablet_copy_client.cc:647] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:03.214447 788 tablet_copy_client.cc:523] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:03.215206 28661 consensus_queue.cc:237] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:03.215929 788 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:03.216688 30476 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:03.217096 32385 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.218793 28398 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:03.219240 31414 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.221118 31414 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.223340 30477 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.223956 28254 catalog_manager.cc:5484] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.227327 28398 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.238597 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.245218 788 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:03.245572 788 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:03.245783 788 ts_tablet_manager.cc:1367] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.003s
I20240502 08:00:03.246414 788 raft_consensus.cc:348] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.246556 788 raft_consensus.cc:729] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:03.246727 788 consensus_queue.cc:260] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.247028 788 ts_tablet_manager.cc:1398] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:03.247484 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-6a40382745f84f0eadf9e0ee2e057289 received from {username='slave'} at 127.16.36.200:55771
I20240502 08:00:03.247567 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-6a40382745f84f0eadf9e0ee2e057289 on tablet 6a40382745f84f0eadf9e0ee2e057289 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:03.275461 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.281328 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.281400 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:03.283082 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.285202 28378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:03.305428 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet d08ad2a981fc4c13bc89d22feea7d556 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:03.305959 843 tablet_replica.cc:330] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:03.306376 843 raft_consensus.cc:2229] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:03.306741 843 raft_consensus.cc:2258] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:03.307421 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.307917 843 ts_tablet_manager.cc:1875] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:03.314886 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.323949 28398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7"
dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38786
I20240502 08:00:03.324132 28398 raft_consensus.cc:595] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:03.339634 843 ts_tablet_manager.cc:1888] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:03.339797 843 log.cc:1196] T d08ad2a981fc4c13bc89d22feea7d556 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/d08ad2a981fc4c13bc89d22feea7d556
I20240502 08:00:03.340826 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet d08ad2a981fc4c13bc89d22feea7d556 (table rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]) successfully deleted
I20240502 08:00:03.401517 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.403962 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.410741 28378 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:03.420809 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.448315 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.459379 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:03.484392 31538 raft_consensus.cc:979] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: : Instructing follower ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to start an election
I20240502 08:00:03.484577 31538 raft_consensus.cc:1067] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Signalling peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to start an election
I20240502 08:00:03.485379 28661 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7"
dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
from {username='slave'} at 127.16.36.193:45489
I20240502 08:00:03.485548 28661 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:03.485610 28661 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.485608 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.486841 28661 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.487320 28661 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 2 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
I20240502 08:00:03.488153 30608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "e5e9553dabc94f218f6427c1685ffc38"
I20240502 08:00:03.488281 30608 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.488596 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 08:00:03.488726 28398 raft_consensus.cc:3032] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:03.488767 28398 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 08:00:03.488952 28398 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.489141 28398 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.489434 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.490274 28398 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 2.
I20240502 08:00:03.491091 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:03.491235 28530 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:03.492354 28530 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 2.
I20240502 08:00:03.493126 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.493467 28594 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 08:00:03.493691 31414 raft_consensus.cc:2781] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:03.493768 31414 raft_consensus.cc:686] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 LEADER]: Becoming Leader. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 08:00:03.493922 31414 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:03.496284 30608 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed3ca7e829bc42b5a8e9a7d1bdcc5a44 in term 2.
I20240502 08:00:03.497910 28252 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: term changed from 1 to 2, leader changed from 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) to ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195). New cstate: current_term: 2 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:03.500368 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.503921 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.504945 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.522094 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.523545 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:03.527310 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.538436 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.541760 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.550895 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.556881 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.558585 30608 raft_consensus.cc:1203] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:00:03.588840 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.589558 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:03.589947 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.603104 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.628057 28530 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:03.629432 30345 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:03.629448 28398 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:03.629968 31798 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.630559 31798 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.631759 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.637775 31798 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:03.641053 28398 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:03.640795 30345 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:03.641108 28252 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:03.645151 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7e7dadff0a4cd08deb043a37a54424 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:03.680812 993 tablet_replica.cc:330] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:03.681249 993 raft_consensus.cc:2229] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:03.681484 993 raft_consensus.cc:2258] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:03.682735 993 ts_tablet_manager.cc:1875] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:03.691447 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.699317 993 ts_tablet_manager.cc:1888] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:03.699467 993 log.cc:1196] T 9c7e7dadff0a4cd08deb043a37a54424 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/9c7e7dadff0a4cd08deb043a37a54424
I20240502 08:00:03.700150 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 9c7e7dadff0a4cd08deb043a37a54424 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:03.712911 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.715723 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.716184 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:03.716640 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.724179 28792 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:03.724668 32399 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.742627 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.756927 30608 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:03.757653 32399 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.759177 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.759699 28661 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:03.761149 32399 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.777691 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.800431 28530 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:03.802536 30214 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:03.803876 30477 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:03.804011 31420 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:03.804785 28792 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:03.805408 991 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:03.805773 991 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:03.816926 991 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.817518 30477 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.819285 30214 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.822424 28250 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.827313 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:03.827651 993 tablet_replica.cc:330] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:03.827816 993 raft_consensus.cc:2229] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:03.827986 993 raft_consensus.cc:2258] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:03.828078 28790 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:03.828644 993 ts_tablet_manager.cc:1875] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:03.831008 993 ts_tablet_manager.cc:1888] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:03.831086 993 log.cc:1196] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/04d358245cdf4b2ebb0b5005f75cf1e0
I20240502 08:00:03.833834 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 04d358245cdf4b2ebb0b5005f75cf1e0 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:03.858237 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.865903 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:03.866405 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.876879 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.878504 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.893476 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.901978 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.913676 30608 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:03.914178 31414 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.941427 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.960198 28530 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:03.961097 31414 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:03.966017 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.971275 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.972864 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:03.973141 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:03.975589 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:03.982089 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:03.999902 28398 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:04.000828 32385 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.007483 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.024911 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.078598 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.090548 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.092751 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:04.095481 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.097220 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.114192 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.115855 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.118418 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.141292 28530 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:04.142720 30477 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:04.142979 28660 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:04.143460 31798 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.150239 31798 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.161108 31420 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.166025 30477 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.169386 28252 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.171553 28660 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.174189 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbf7921ae84476ab53fbd4707f463eb with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:04.174727 1274 tablet_replica.cc:330] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:04.175156 1274 raft_consensus.cc:2229] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:04.175488 1274 raft_consensus.cc:2258] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:04.176788 1274 ts_tablet_manager.cc:1875] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:04.179690 1274 ts_tablet_manager.cc:1888] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:04.179893 1274 log.cc:1196] T 7dbf7921ae84476ab53fbd4707f463eb P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/7dbf7921ae84476ab53fbd4707f463eb
I20240502 08:00:04.180617 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 7dbf7921ae84476ab53fbd4707f463eb (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
I20240502 08:00:04.205869 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.208846 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.209178 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.221421 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:04.226850 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.232962 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.249522 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.251895 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.314028 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.314674 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.320070 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:04.323432 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.336406 28660 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:04.338135 28790 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:04.338681 32385 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.340112 30214 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:04.341298 32385 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.346458 1188 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:04.349285 28790 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:04.349529 28252 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:04.351559 30214 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:04.356122 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.358335 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 698f494a22d34e659766b97cb24d0293 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:04.363353 28790 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:04.365170 30606 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:04.365408 28530 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:04.365818 472 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.365969 30477 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:04.366170 472 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.366778 1377 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.370029 472 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.370530 28530 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.373927 30606 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.375720 28256 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.381045 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a40382745f84f0eadf9e0ee2e057289 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:04.395963 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.402953 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.404665 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.406797 1375 tablet_replica.cc:330] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:04.407231 1375 raft_consensus.cc:2229] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:04.407531 1375 raft_consensus.cc:2258] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:04.408815 1375 ts_tablet_manager.cc:1875] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:04.411502 1375 ts_tablet_manager.cc:1888] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:04.425664 30477 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.442975 1375 log.cc:1196] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/698f494a22d34e659766b97cb24d0293
I20240502 08:00:04.443807 1375 tablet_replica.cc:330] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:04.444041 1375 raft_consensus.cc:2229] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:04.444317 1375 raft_consensus.cc:2258] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:04.444586 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 698f494a22d34e659766b97cb24d0293 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:04.445315 1375 ts_tablet_manager.cc:1875] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:04.447667 1375 ts_tablet_manager.cc:1888] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:04.447989 1375 log.cc:1196] T 6a40382745f84f0eadf9e0ee2e057289 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/6a40382745f84f0eadf9e0ee2e057289
I20240502 08:00:04.448742 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 6a40382745f84f0eadf9e0ee2e057289 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:04.479502 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.490190 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.491292 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.499886 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.508646 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.508985 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.523690 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:04.528201 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.530395 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:04.530647 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.531196 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.547915 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.559849 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.588112 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.600041 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.600860 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.603058 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.610708 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.634392 28509 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.639974 28771 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:04.648041 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.649046 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.650861 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.674799 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.715682 28398 consensus_queue.cc:237] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:04.718194 30214 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:04.718647 30476 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:04.718896 28660 raft_consensus.cc:1261] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:04.718835 31538 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.719085 1468 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.719573 1468 consensus_queue.cc:1035] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.721915 31538 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.722493 28660 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.725135 28254 catalog_manager.cc:5484] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.725198 30214 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.731824 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet c47d608895f94a2c82e695ca16677abf with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:04.741178 1527 tablet_replica.cc:330] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:04.742575 1527 raft_consensus.cc:2229] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:04.742884 1527 raft_consensus.cc:2258] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:04.744514 1527 ts_tablet_manager.cc:1875] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:04.748565 30476 raft_consensus.cc:2932] T c47d608895f94a2c82e695ca16677abf P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:04.752732 28530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0"
dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35640
I20240502 08:00:04.752916 28530 raft_consensus.cc:595] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:04.756708 28398 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:04.758607 28660 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:04.759264 1518 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.759927 28790 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:04.760573 1518 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:04.761101 1527 ts_tablet_manager.cc:1888] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:04.761247 1527 log.cc:1196] T c47d608895f94a2c82e695ca16677abf P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/c47d608895f94a2c82e695ca16677abf
I20240502 08:00:04.761934 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet c47d608895f94a2c82e695ca16677abf (table rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]) successfully deleted
I20240502 08:00:04.762759 1518 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:04.765429 28660 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:04.766100 28256 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:04.766105 28790 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:04.788210 30588 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f7c358ace742649df52380939ece9d with delete_type TABLET_DATA_TOMBSTONED (TS e5e9553dabc94f218f6427c1685ffc38 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43060
I20240502 08:00:04.795253 991 raft_consensus.cc:979] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: : Instructing follower de423605477a4819a9569821f59f03e4 to start an election
I20240502 08:00:04.795357 991 raft_consensus.cc:1067] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Signalling peer de423605477a4819a9569821f59f03e4 to start an election
I20240502 08:00:04.796165 28790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0"
dest_uuid: "de423605477a4819a9569821f59f03e4"
from {username='slave'} at 127.16.36.194:51335
I20240502 08:00:04.796322 28790 raft_consensus.cc:482] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:04.796372 28790 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:04.797448 28790 raft_consensus.cc:504] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:04.797888 28790 leader_election.cc:288] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 2 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:04.798528 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:04.798741 30214 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:04.800004 30214 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:04.801084 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:04.801221 28530 raft_consensus.cc:3032] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:04.801263 28530 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:04.801431 28530 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:04.801640 28530 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:04.802138 1579 tablet_replica.cc:330] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: stopping tablet replica
I20240502 08:00:04.802506 1579 raft_consensus.cc:2229] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:04.802749 28530 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:04.803277 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0499a3f3e6bd4069bf992e32f1abfdb0" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 08:00:04.803421 28660 raft_consensus.cc:3037] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:04.804028 1579 raft_consensus.cc:2258] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:04.806772 28725 leader_election.cc:302] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:04.807032 1225 raft_consensus.cc:2781] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:04.807132 1225 raft_consensus.cc:686] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 08:00:04.807312 1225 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:04.807155 1579 ts_tablet_manager.cc:1875] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:04.810067 28660 raft_consensus.cc:2454] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:04.811261 1579 ts_tablet_manager.cc:1888] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:00:04.811365 1579 log.cc:1196] T 57f7c358ace742649df52380939ece9d P e5e9553dabc94f218f6427c1685ffc38: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-7/wal/wals/57f7c358ace742649df52380939ece9d
I20240502 08:00:04.811269 28256 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 1 to 2, leader changed from c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:04.811975 28241 catalog_manager.cc:4830] TS e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): tablet 57f7c358ace742649df52380939ece9d (table rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]) successfully deleted
I20240502 08:00:04.818799 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.833444 28509 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:04.836390 28397 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:04.838088 28530 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:04.838179 30606 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:04.838492 1468 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.838680 31538 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:04.843250 1468 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.845711 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:04.846905 30606 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.846576 28256 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.849767 28530 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:04.850265 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:04.854525 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 3288b438832e488e8257227cb8934565 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:04.855358 1527 tablet_replica.cc:330] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:04.855502 1527 raft_consensus.cc:2229] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:04.855638 1527 raft_consensus.cc:2258] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:04.856228 1527 ts_tablet_manager.cc:1875] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:04.859804 1527 ts_tablet_manager.cc:1888] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:04.859885 1527 log.cc:1196] T 3288b438832e488e8257227cb8934565 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/3288b438832e488e8257227cb8934565
I20240502 08:00:04.860447 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 3288b438832e488e8257227cb8934565 (table rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]) successfully deleted
I20240502 08:00:04.861879 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.862290 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.870656 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.885260 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.964246 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.964259 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.964818 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:04.971774 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:04.985757 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:05.009110 28509 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:05.032100 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.032691 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.040745 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.051653 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.066526 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.075086 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.075196 28509 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:05.076588 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:05.085678 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.094600 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.126693 28660 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:05.128305 28530 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.128826 32385 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:05.129999 30345 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.130612 32385 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:05.130709 30214 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.131290 32385 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:05.148783 32385 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.153437 28250 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.154619 30345 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.157632 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet afc9c4d9dd5b4c039686ccd918616859 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:05.157876 1527 tablet_replica.cc:330] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:05.157721 30214 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.157999 1527 raft_consensus.cc:2229] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:05.158139 1527 raft_consensus.cc:2258] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:05.158702 1527 ts_tablet_manager.cc:1875] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:05.162303 28530 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.163441 1527 ts_tablet_manager.cc:1888] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:05.163532 1527 log.cc:1196] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/afc9c4d9dd5b4c039686ccd918616859
I20240502 08:00:05.164139 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet afc9c4d9dd5b4c039686ccd918616859 (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
I20240502 08:00:05.173588 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.177165 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.177227 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.182994 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.184747 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:05.195293 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.203545 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.221621 28509 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:05.299050 28660 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:05.300912 30214 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.301390 1736 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:05.303436 1736 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.305822 30214 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.306985 28254 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.312197 28660 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:05.312508 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 698f494a22d34e659766b97cb24d0293 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:05.312743 1527 tablet_replica.cc:330] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:05.312893 1527 raft_consensus.cc:2229] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:05.313047 1527 raft_consensus.cc:2258] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:05.314725 30214 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:00:05.314963 28597 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): Couldn't send request to peer de423605477a4819a9569821f59f03e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:05.315208 1754 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:05.315522 1527 ts_tablet_manager.cc:1875] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:05.317490 1754 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:05.318114 1527 ts_tablet_manager.cc:1888] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:05.318183 1527 log.cc:1196] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/698f494a22d34e659766b97cb24d0293
I20240502 08:00:05.318905 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 698f494a22d34e659766b97cb24d0293 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:05.320695 28243 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 698f494a22d34e659766b97cb24d0293 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:05.322085 28249 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:05.322202 30214 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:05.327014 28530 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:05.327718 1377 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:05.342773 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.348830 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:05.351135 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.359290 30214 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:05.359776 1377 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:05.363898 28660 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 08:00:05.365108 1377 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:05.366066 28509 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:05.373441 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.399657 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.404433 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.430086 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.439116 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.440920 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.450592 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.452219 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.458884 28509 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:05.466462 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.473582 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:05.491114 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.554461 28530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0"
dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35728
I20240502 08:00:05.554656 28530 raft_consensus.cc:595] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:05.559592 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.562786 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.570627 31798 raft_consensus.cc:979] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: : Instructing follower de423605477a4819a9569821f59f03e4 to start an election
I20240502 08:00:05.570964 31798 raft_consensus.cc:1067] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Signalling peer de423605477a4819a9569821f59f03e4 to start an election
I20240502 08:00:05.572820 28792 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0"
dest_uuid: "de423605477a4819a9569821f59f03e4"
from {username='slave'} at 127.16.36.194:51335
I20240502 08:00:05.572979 28792 raft_consensus.cc:482] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:05.573055 28792 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:05.574368 28792 raft_consensus.cc:504] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:05.574883 28792 leader_election.cc:288] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 2 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407), 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589)
I20240502 08:00:05.575222 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:05.575345 28530 raft_consensus.cc:3032] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:05.575384 28530 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:05.575520 28530 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:05.575727 28530 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:05.576237 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:05.576368 30214 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:05.576937 28530 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:05.577540 30214 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:05.577793 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.578054 30476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "29126d5aee924f81bdd47cc7153420a1"
I20240502 08:00:05.578238 30476 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:05.578366 28728 leader_election.cc:302] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:05.578557 1901 raft_consensus.cc:2781] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:05.578634 1901 raft_consensus.cc:686] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 08:00:05.578796 1901 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:05.579571 30476 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:05.583109 28253 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 1 to 2, leader changed from c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:05.605851 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.610251 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.613806 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.618408 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.626847 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:05.635095 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.638355 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:05.647809 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.660539 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.669413 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.675698 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.689167 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.690585 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.793365 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.796610 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.798333 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.812340 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:05.816591 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.830832 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.840798 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.843938 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.851802 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.856855 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:05.884956 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.886313 28660 consensus_queue.cc:237] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:05.889305 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.889542 28530 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.890575 30214 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.892961 30345 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:05.894400 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.904651 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.909416 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:05.890105 2120 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20240502 08:00:05.914877 1753 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:05.916371 1746 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:05.917244 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:05.918020 1753 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.918475 30214 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.920336 28530 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.923211 28253 catalog_manager.cc:5484] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.926107 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:05.930994 2129 tablet_replica.cc:330] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:05.931263 2129 raft_consensus.cc:2229] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:05.931473 2129 raft_consensus.cc:2258] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:05.931882 30345 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:05.948520 2129 ts_tablet_manager.cc:1875] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:05.959197 30214 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:05.959651 1377 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:05.969506 2129 ts_tablet_manager.cc:1888] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:05.969704 2129 log.cc:1196] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/de90ee4ef90e4cb69ec37d7d9f0c86ef
I20240502 08:00:05.970367 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet de90ee4ef90e4cb69ec37d7d9f0c86ef (table rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]) successfully deleted
I20240502 08:00:05.987218 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.987881 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:05.990530 28530 consensus_queue.cc:237] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:05.992060 30345 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:05.992261 28398 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:05.992563 1995 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:05.992908 31798 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:05.995021 31798 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:05.997102 30345 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.004607 28253 catalog_manager.cc:5484] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.008595 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 167b5f3b818048af81b1350839ebe489 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:06.009600 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:06.010254 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.012948 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.015493 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.023430 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.029134 30476 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:06.029677 1377 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:06.047406 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.051748 28530 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:06.052158 1901 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:06.053825 2175 tablet_replica.cc:330] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:06.053964 2175 raft_consensus.cc:2229] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:06.054108 2175 raft_consensus.cc:2258] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:06.054636 2175 ts_tablet_manager.cc:1875] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:06.065220 2175 ts_tablet_manager.cc:1888] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:06.065419 2175 log.cc:1196] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/167b5f3b818048af81b1350839ebe489
I20240502 08:00:06.074942 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 167b5f3b818048af81b1350839ebe489 (table rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]) successfully deleted
I20240502 08:00:06.074903 28398 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.097474 28530 consensus_queue.cc:237] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:06.099336 28660 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.099431 30345 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.099999 31798 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.100266 30608 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.100396 31798 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.100817 1995 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.104028 1995 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.106878 30345 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.107057 30608 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.111908 28253 catalog_manager.cc:5484] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.112210 28660 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.114794 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 07461c988fb3402c91cfd9235efbb9b1 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:06.115010 2129 tablet_replica.cc:330] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:06.115129 2129 raft_consensus.cc:2229] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:06.115268 2129 raft_consensus.cc:2258] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:06.115833 2129 ts_tablet_manager.cc:1875] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:06.118144 2129 ts_tablet_manager.cc:1888] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:06.118214 2129 log.cc:1196] T 07461c988fb3402c91cfd9235efbb9b1 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/07461c988fb3402c91cfd9235efbb9b1
I20240502 08:00:06.118819 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 07461c988fb3402c91cfd9235efbb9b1 (table rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]) successfully deleted
I20240502 08:00:06.161165 28660 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:06.164186 30608 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.164572 30476 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.165030 2120 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.165923 28792 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.165906 2120 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.167310 2121 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.176432 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.178090 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.194962 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.200284 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.219476 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.219939 2120 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:06.222927 30608 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:06.225032 30477 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:06.226564 28253 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:06.229127 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.233445 2323 ts_tablet_manager.cc:902] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:06.235441 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet f98044a1730e4a04a7088199d4f0850e with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:06.235672 2129 tablet_replica.cc:330] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:06.235798 2129 raft_consensus.cc:2229] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:06.235949 2129 raft_consensus.cc:2258] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:06.236482 2129 ts_tablet_manager.cc:1875] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:06.238870 2129 ts_tablet_manager.cc:1888] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:06.238955 2129 log.cc:1196] T f98044a1730e4a04a7088199d4f0850e P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/f98044a1730e4a04a7088199d4f0850e
I20240502 08:00:06.239907 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.240155 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet f98044a1730e4a04a7088199d4f0850e (table rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]) successfully deleted
I20240502 08:00:06.241638 28792 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:06.243826 2323 tablet_copy_client.cc:308] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:06.244422 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 698f494a22d34e659766b97cb24d0293 from peer de423605477a4819a9569821f59f03e4 ({username='slave'} at 127.16.36.196:59091)
I20240502 08:00:06.244560 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 698f494a22d34e659766b97cb24d0293 from peer de423605477a4819a9569821f59f03e4 at {username='slave'} at 127.16.36.196:59091: session id = de423605477a4819a9569821f59f03e4-698f494a22d34e659766b97cb24d0293
I20240502 08:00:06.245441 28681 tablet_copy_source_session.cc:205] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:06.246426 2323 ts_tablet_manager.cc:1875] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:06.248925 2323 ts_tablet_manager.cc:1888] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:00:06.249255 2323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:06.250777 2323 tablet_copy_client.cc:783] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:06.251101 2323 tablet_copy_client.cc:647] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:06.256004 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:06.264914 2323 tablet_copy_client.cc:523] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:06.267473 2323 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 08:00:06.285261 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.285392 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.293568 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.307673 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:06.320396 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.327493 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.344103 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.345607 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.348807 2323 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:06.349220 2323 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Bootstrap complete.
I20240502 08:00:06.349407 2323 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.082s user 0.024s sys 0.005s
I20240502 08:00:06.349928 2323 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:06.350044 2323 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 08:00:06.350252 2323 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:06.352423 2323 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:00:06.353430 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session de423605477a4819a9569821f59f03e4-698f494a22d34e659766b97cb24d0293 received from {username='slave'} at 127.16.36.196:59091
I20240502 08:00:06.353518 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session de423605477a4819a9569821f59f03e4-698f494a22d34e659766b97cb24d0293 on tablet 698f494a22d34e659766b97cb24d0293 with peer de423605477a4819a9569821f59f03e4
I20240502 08:00:06.379716 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.385579 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.414825 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.423216 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.427915 28530 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424"
dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35876
I20240502 08:00:06.428081 28530 raft_consensus.cc:595] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:06.453392 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.459810 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:06.460536 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.462693 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:06.467136 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.476416 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.493527 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.494244 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.500680 30326 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.503767 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.512503 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.540289 991 raft_consensus.cc:979] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: : Instructing follower 6efe17dcca7b4babad4785d19552b699 to start an election
I20240502 08:00:06.540401 991 raft_consensus.cc:1067] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Signalling peer 6efe17dcca7b4babad4785d19552b699 to start an election
I20240502 08:00:06.540920 30345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424"
dest_uuid: "6efe17dcca7b4babad4785d19552b699"
from {username='slave'} at 127.16.36.194:52473
I20240502 08:00:06.541152 30345 raft_consensus.cc:482] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:06.541287 30345 raft_consensus.cc:3037] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:06.542551 30345 raft_consensus.cc:504] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:06.543695 30345 leader_election.cc:288] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [CANDIDATE]: Term 2 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:06.544230 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424" candidate_uuid: "6efe17dcca7b4babad4785d19552b699" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:06.544368 28530 raft_consensus.cc:3032] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:06.544411 28530 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:06.544586 28530 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:06.544740 28530 raft_consensus.cc:3037] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:06.545789 28530 raft_consensus.cc:2454] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efe17dcca7b4babad4785d19552b699 in term 2.
I20240502 08:00:06.547427 30279 leader_election.cc:302] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efe17dcca7b4babad4785d19552b699, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 08:00:06.547864 2139 raft_consensus.cc:2781] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:06.548020 2139 raft_consensus.cc:686] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 2 LEADER]: Becoming Leader. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Running, Role: LEADER
I20240502 08:00:06.548333 2139 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:06.551564 28249 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: term changed from 1 to 2, leader changed from c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) to 6efe17dcca7b4babad4785d19552b699 (127.16.36.198). New cstate: current_term: 2 leader_uuid: "6efe17dcca7b4babad4785d19552b699" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.553156 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c7e7dadff0a4cd08deb043a37a54424" candidate_uuid: "6efe17dcca7b4babad4785d19552b699" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 08:00:06.553310 28398 raft_consensus.cc:3037] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:06.554385 28398 raft_consensus.cc:2454] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efe17dcca7b4babad4785d19552b699 in term 2.
I20240502 08:00:06.556757 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.565492 28790 consensus_queue.cc:237] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:06.568831 30214 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.569833 1901 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.573655 28530 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.574803 2199 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.586153 30345 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:06.587785 2199 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.588735 2199 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.591483 28530 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.594974 30214 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.598373 28254 catalog_manager.cc:5484] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.601301 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bbfb3758e04aeab0df4bbab8380061 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:06.602751 2484 tablet_replica.cc:330] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:06.602936 2484 raft_consensus.cc:2229] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:06.603144 2484 raft_consensus.cc:2258] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:06.603775 2484 ts_tablet_manager.cc:1875] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:06.606348 2484 ts_tablet_manager.cc:1888] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:06.606508 2484 log.cc:1196] T 14bbfb3758e04aeab0df4bbab8380061 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/14bbfb3758e04aeab0df4bbab8380061
I20240502 08:00:06.606953 28790 raft_consensus.cc:1203] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:00:06.607431 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 14bbfb3758e04aeab0df4bbab8380061 (table rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]) successfully deleted
I20240502 08:00:06.615789 30345 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:06.648365 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.668488 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.674340 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.679991 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.682338 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:06.688359 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.693069 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.695734 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.705191 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.706878 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:06.708305 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.716560 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:06.740032 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.741935 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.758780 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.778663 28660 consensus_queue.cc:237] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:06.783502 30214 raft_consensus.cc:1261] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:06.783903 28790 raft_consensus.cc:1261] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:06.784688 1753 consensus_queue.cc:1035] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:06.785028 1753 consensus_queue.cc:1035] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:06.792870 1746 raft_consensus.cc:2932] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:06.793356 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:06.795095 28790 raft_consensus.cc:2932] T dcd68a21ed7b4205a2f0cf75f80df0e6 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:06.797740 28252 catalog_manager.cc:5484] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:06.801000 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet dcd68a21ed7b4205a2f0cf75f80df0e6 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:06.801405 2568 tablet_replica.cc:330] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:06.801574 2568 raft_consensus.cc:2229] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:06.801784 2568 raft_consensus.cc:2258] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:06.802373 2568 ts_tablet_manager.cc:1875] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:06.804589 30214 raft_consensus.cc:2932] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:06.805470 2568 ts_tablet_manager.cc:1888] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:06.805615 2568 log.cc:1196] T dcd68a21ed7b4205a2f0cf75f80df0e6 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/dcd68a21ed7b4205a2f0cf75f80df0e6
I20240502 08:00:06.810541 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet dcd68a21ed7b4205a2f0cf75f80df0e6 (table rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]) successfully deleted
I20240502 08:00:06.815246 28790 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:06.826027 30476 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:06.826507 2457 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:06.826767 30214 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:06.827359 2457 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:06.828963 2457 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:06.835990 28252 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:06.836290 30476 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:06.843709 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:06.843731 30215 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:06.853310 2608 tablet_replica.cc:330] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:06.853574 2608 raft_consensus.cc:2229] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:06.853796 2608 raft_consensus.cc:2258] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:06.854743 2608 ts_tablet_manager.cc:1875] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:06.863695 2608 ts_tablet_manager.cc:1888] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:00:06.863886 2608 log.cc:1196] T 04d358245cdf4b2ebb0b5005f75cf1e0 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/04d358245cdf4b2ebb0b5005f75cf1e0
I20240502 08:00:06.864567 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 04d358245cdf4b2ebb0b5005f75cf1e0 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:06.888564 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.892915 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:06.902194 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.911883 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:06.913733 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:06.925119 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:06.937245 28398 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:06.938338 2449 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.944384 28530 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:06.944866 2139 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:06.956537 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:06.964504 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.040344 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.056425 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.063400 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.084757 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.096499 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.101368 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.103459 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.109849 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.114925 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.117126 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.131254 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.135819 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.143116 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.154192 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.156116 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.159071 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.160090 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.176589 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.183341 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.192950 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.202435 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.215313 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.223507 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.241406 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.266309 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.280612 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.282508 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.283247 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.284157 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.299894 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.300122 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.303552 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.328994 28790 consensus_queue.cc:237] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:07.330580 30606 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:07.331271 28530 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:07.331634 1901 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:07.334647 1901 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:07.349857 1901 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.352761 28530 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.353049 28253 catalog_manager.cc:5484] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.355590 30606 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.371868 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet fddc382ab3164ccdaa86f802c83da8be with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:07.376977 2835 tablet_replica.cc:330] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:07.377310 2835 raft_consensus.cc:2229] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:07.377628 2835 raft_consensus.cc:2258] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:07.378319 2835 ts_tablet_manager.cc:1875] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:07.380532 2835 ts_tablet_manager.cc:1888] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:07.380678 2835 log.cc:1196] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/fddc382ab3164ccdaa86f802c83da8be
I20240502 08:00:07.381754 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet fddc382ab3164ccdaa86f802c83da8be (table rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]) successfully deleted
I20240502 08:00:07.388664 30345 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:07.390558 28398 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:07.391830 2139 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:07.395522 2139 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:07.398574 28255 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: config changed from index 5 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "6efe17dcca7b4babad4785d19552b699" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:07.399389 28397 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:07.402285 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c7e7dadff0a4cd08deb043a37a54424 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:07.403131 2870 tablet_replica.cc:330] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:07.403340 2870 raft_consensus.cc:2229] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:07.403525 2870 raft_consensus.cc:2258] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:07.404350 2870 ts_tablet_manager.cc:1875] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:07.406946 2870 ts_tablet_manager.cc:1888] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:00:07.407406 2870 log.cc:1196] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c7e7dadff0a4cd08deb043a37a54424
I20240502 08:00:07.408102 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 9c7e7dadff0a4cd08deb043a37a54424 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:07.409936 28790 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:07.411406 30476 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:07.411515 28529 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:07.411958 2199 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:07.412343 2813 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:07.414441 2813 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:07.416183 30476 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:07.417822 28255 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:07.423489 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d939e5035541a4b003c5d276493d5c with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:07.423679 30345 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:07.425217 28529 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:07.425386 2905 tablet_replica.cc:330] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:07.425527 2905 raft_consensus.cc:2229] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:07.425704 2905 raft_consensus.cc:2258] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:07.426177 2905 ts_tablet_manager.cc:1875] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:07.429463 2905 ts_tablet_manager.cc:1888] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:07.431977 2905 log.cc:1196] T 21d939e5035541a4b003c5d276493d5c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/21d939e5035541a4b003c5d276493d5c
I20240502 08:00:07.432236 28398 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:07.433194 28397 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:07.433656 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 21d939e5035541a4b003c5d276493d5c (table rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]) successfully deleted
I20240502 08:00:07.433912 30606 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:07.434475 28529 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:07.434566 1518 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:07.435946 2322 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:07.436866 2139 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:07.439042 2139 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:07.440845 28243 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9c7e7dadff0a4cd08deb043a37a54424 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:07.442142 28251 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: config changed from index 7 to 8, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 2 leader_uuid: "6efe17dcca7b4babad4785d19552b699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
W20240502 08:00:07.442615 30282 consensus_peers.cc:485] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c7e7dadff0a4cd08deb043a37a54424. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:07.442731 28397 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:07.444409 2322 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.445705 28529 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.448724 28251 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.452267 30606 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:07.454090 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 3598dbda342f49aea2384b5beb17d7a1 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:07.458045 28790 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:07.459848 30215 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:07.460296 2813 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:07.461563 28529 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:07.462265 2813 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:07.462587 2933 tablet_replica.cc:330] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:07.462720 2933 raft_consensus.cc:2229] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:07.462867 2933 raft_consensus.cc:2258] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:07.463402 2933 ts_tablet_manager.cc:1875] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:07.463778 2199 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:07.464370 30215 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:07.467016 28255 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 4 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:07.467554 28529 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 4 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:07.471581 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:07.471829 2905 tablet_replica.cc:330] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:07.471972 2905 raft_consensus.cc:2229] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:07.472136 2905 raft_consensus.cc:2258] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:07.472842 2905 ts_tablet_manager.cc:1875] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:07.475750 2905 ts_tablet_manager.cc:1888] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.5
I20240502 08:00:07.476001 2905 log.cc:1196] T 0499a3f3e6bd4069bf992e32f1abfdb0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/0499a3f3e6bd4069bf992e32f1abfdb0
I20240502 08:00:07.476660 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 0499a3f3e6bd4069bf992e32f1abfdb0 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:07.478430 2933 ts_tablet_manager.cc:1888] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:07.478617 2933 log.cc:1196] T 3598dbda342f49aea2384b5beb17d7a1 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/3598dbda342f49aea2384b5beb17d7a1
I20240502 08:00:07.479231 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 3598dbda342f49aea2384b5beb17d7a1 (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
I20240502 08:00:07.558514 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.558828 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.559674 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.568239 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.571470 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.591217 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.596642 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.629015 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.653337 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.655853 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.657441 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.662853 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.668761 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.669520 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.718145 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.729893 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.730744 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.735232 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.747875 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.747891 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.771457 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.776854 28660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7"
dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56046
I20240502 08:00:07.777037 28660 raft_consensus.cc:595] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 LEADER]: Received request to transfer leadership
I20240502 08:00:07.780840 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.782110 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.803546 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.804150 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.809036 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.826639 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.853531 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.855989 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.864125 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.878407 3139 ts_tablet_manager.cc:902] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 08:00:07.879230 3139 tablet_copy_client.cc:308] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.198:45421
I20240502 08:00:07.885653 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.885704 30366 tablet_copy_service.cc:140] P 6efe17dcca7b4babad4785d19552b699: Received BeginTabletCopySession request for tablet 9c7e7dadff0a4cd08deb043a37a54424 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:33809)
I20240502 08:00:07.885834 30366 tablet_copy_service.cc:161] P 6efe17dcca7b4babad4785d19552b699: Beginning new tablet copy session on tablet 9c7e7dadff0a4cd08deb043a37a54424 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:33809: session id = 6c68c916f39a45f697df7eaf8973225a-9c7e7dadff0a4cd08deb043a37a54424
I20240502 08:00:07.887308 30366 tablet_copy_source_session.cc:205] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:07.890161 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.901340 3139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7e7dadff0a4cd08deb043a37a54424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:07.909166 3139 tablet_copy_client.cc:783] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:07.909489 3139 tablet_copy_client.cc:647] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:07.911101 3139 tablet_copy_client.cc:523] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:07.912628 3139 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:07.941715 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.950107 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:07.953954 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.956832 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:07.957617 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:07.961251 28639 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 08:00:07.964766 3139 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:07.965128 3139 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:07.965673 3139 ts_tablet_manager.cc:1367] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.053s user 0.026s sys 0.004s
I20240502 08:00:07.966252 3139 raft_consensus.cc:348] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:07.966387 3139 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:07.975924 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:07.966555 3139 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:07.980398 3139 ts_tablet_manager.cc:1398] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240502 08:00:07.984630 30366 tablet_copy_service.cc:342] P 6efe17dcca7b4babad4785d19552b699: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-9c7e7dadff0a4cd08deb043a37a54424 received from {username='slave'} at 127.16.36.197:33809
I20240502 08:00:07.984742 30366 tablet_copy_service.cc:433] P 6efe17dcca7b4babad4785d19552b699: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-9c7e7dadff0a4cd08deb043a37a54424 on tablet 9c7e7dadff0a4cd08deb043a37a54424 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:07.993813 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.031185 28660 consensus_queue.cc:237] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:08.032888 30215 raft_consensus.cc:1261] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:08.033488 28397 raft_consensus.cc:1261] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:08.033565 1746 consensus_queue.cc:1035] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:08.034360 1746 consensus_queue.cc:1035] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:08.039422 1746 raft_consensus.cc:2932] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.042318 28397 raft_consensus.cc:2932] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.045652 28249 catalog_manager.cc:5484] T df20642aef7748ff8962b3095e1ab56c P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.049908 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet df20642aef7748ff8962b3095e1ab56c with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:08.051759 3242 tablet_replica.cc:330] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:08.051932 3242 raft_consensus.cc:2229] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:08.052098 3242 raft_consensus.cc:2258] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:08.054621 3242 ts_tablet_manager.cc:1875] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:08.056842 28530 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.058661 30606 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.059453 30476 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.059654 28397 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3288b438832e488e8257227cb8934565"
dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39282
I20240502 08:00:08.059800 28397 raft_consensus.cc:595] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:08.060000 991 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.060405 991 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.058497 28790 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.067880 2681 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.068873 30215 raft_consensus.cc:2932] T df20642aef7748ff8962b3095e1ab56c P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.073768 3242 ts_tablet_manager.cc:1888] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:08.073953 3242 log.cc:1196] T df20642aef7748ff8962b3095e1ab56c P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/df20642aef7748ff8962b3095e1ab56c
I20240502 08:00:08.074584 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet df20642aef7748ff8962b3095e1ab56c (table rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]) successfully deleted
I20240502 08:00:08.076345 28660 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.080839 30476 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:08.081781 30606 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:08.082578 3219 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:08.087194 1753 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:08.087400 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.087924 30476 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.078400 2681 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.092777 30606 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.093077 28257 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.094857 28378 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.095873 1746 raft_consensus.cc:979] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: : Instructing follower e5e9553dabc94f218f6427c1685ffc38 to start an election
I20240502 08:00:08.096401 1746 raft_consensus.cc:1067] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 LEADER]: Signalling peer e5e9553dabc94f218f6427c1685ffc38 to start an election
I20240502 08:00:08.097528 30608 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7"
dest_uuid: "e5e9553dabc94f218f6427c1685ffc38"
from {username='slave'} at 127.16.36.195:37125
I20240502 08:00:08.097731 30608 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:08.097810 30608 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:08.098866 30608 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.100086 30608 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [CANDIDATE]: Term 3 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:08.100781 28530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "e5e9553dabc94f218f6427c1685ffc38" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:08.101264 28530 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:08.102210 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "e5e9553dabc94f218f6427c1685ffc38" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 08:00:08.102347 28397 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:08.102352 28530 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e9553dabc94f218f6427c1685ffc38 in term 3.
I20240502 08:00:08.103435 28397 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e9553dabc94f218f6427c1685ffc38 in term 3.
I20240502 08:00:08.104192 30542 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e, e5e9553dabc94f218f6427c1685ffc38; no voters:
I20240502 08:00:08.104564 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "e5e9553dabc94f218f6427c1685ffc38" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 08:00:08.104699 28660 raft_consensus.cc:3032] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:00:08.104766 28660 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 08:00:08.104914 28660 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.105432 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.105554 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1cd0ec72d1429f9621d217b1bf845a with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:08.105108 28660 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:08.109767 28660 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e9553dabc94f218f6427c1685ffc38 in term 3.
I20240502 08:00:08.111409 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:08.120787 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.120823 28791 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.121264 1753 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.123971 30476 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.127066 28257 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.128419 30606 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.131606 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet f98044a1730e4a04a7088199d4f0850e with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:08.131855 3242 tablet_replica.cc:330] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:08.132000 3242 raft_consensus.cc:2229] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:08.132179 3242 raft_consensus.cc:2258] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:08.132679 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.132802 3242 ts_tablet_manager.cc:1875] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:08.135301 3242 ts_tablet_manager.cc:1888] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:00:08.135381 3242 log.cc:1196] T f98044a1730e4a04a7088199d4f0850e P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/f98044a1730e4a04a7088199d4f0850e
I20240502 08:00:08.137346 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet f98044a1730e4a04a7088199d4f0850e (table rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]) successfully deleted
I20240502 08:00:08.139093 30588 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.148578 3274 raft_consensus.cc:2781] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:00:08.148784 3274 raft_consensus.cc:686] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 3 LEADER]: Becoming Leader. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Running, Role: LEADER
I20240502 08:00:08.149439 3274 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.154578 3276 tablet_replica.cc:330] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:08.154811 3276 raft_consensus.cc:2229] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:08.155031 3276 raft_consensus.cc:2258] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:08.157205 28249 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 reported cstate change: term changed from 2 to 3, leader changed from ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) to e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200). New cstate: current_term: 3 leader_uuid: "e5e9553dabc94f218f6427c1685ffc38" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.156855 3276 ts_tablet_manager.cc:1875] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:08.163756 3276 ts_tablet_manager.cc:1888] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:08.163944 3276 log.cc:1196] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e1cd0ec72d1429f9621d217b1bf845a
I20240502 08:00:08.164685 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 9e1cd0ec72d1429f9621d217b1bf845a (table rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]) successfully deleted
I20240502 08:00:08.165606 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.218305 3193 raft_consensus.cc:979] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: : Instructing follower c39bcb719db947b8afb0bb141e1f7a4e to start an election
I20240502 08:00:08.218417 3193 raft_consensus.cc:1067] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Signalling peer c39bcb719db947b8afb0bb141e1f7a4e to start an election
I20240502 08:00:08.219092 28530 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3288b438832e488e8257227cb8934565"
dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
from {username='slave'} at 127.16.36.193:47939
I20240502 08:00:08.219269 28530 raft_consensus.cc:482] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:08.219336 28530 raft_consensus.cc:3037] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.220521 28530 raft_consensus.cc:504] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.221030 28530 leader_election.cc:288] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 2 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
I20240502 08:00:08.221541 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3288b438832e488e8257227cb8934565" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 08:00:08.221700 28397 raft_consensus.cc:3032] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:08.221761 28397 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 08:00:08.221930 28397 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.222121 28397 raft_consensus.cc:3037] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.223282 28397 raft_consensus.cc:2454] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 2.
I20240502 08:00:08.223635 28464 leader_election.cc:302] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71922d97e1a4414698ecd14a7b2dedec, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 08:00:08.223826 3245 raft_consensus.cc:2781] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:08.223902 3245 raft_consensus.cc:686] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:08.224046 3245 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:08.225425 30606 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3288b438832e488e8257227cb8934565" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e5e9553dabc94f218f6427c1685ffc38"
I20240502 08:00:08.225569 30606 raft_consensus.cc:3037] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.226701 30606 raft_consensus.cc:2454] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 2.
I20240502 08:00:08.231222 28257 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 1 to 2, leader changed from 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:08.259567 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.260998 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.270538 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.273200 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.284627 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:08.294283 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.331449 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:08.337472 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.340824 30215 raft_consensus.cc:1203] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:00:08.397505 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.406291 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.407042 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.408814 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.409660 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:08.409868 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.414245 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.451198 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.454151 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:08.458467 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:08.474062 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.484054 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.493945 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.494879 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.496021 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:08.506623 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.522145 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:08.522924 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.534299 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.542656 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.543507 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:08.566645 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.573969 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.582034 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.613834 28397 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 08:00:08.614542 3275 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.648041 28791 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:08.649713 30476 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.650275 3186 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.655678 28529 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 08:00:08.657305 3275 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.664168 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.666504 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.666654 30606 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:08.667747 3245 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.678429 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.684337 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:08.684914 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.685364 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.686527 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:08.693603 3186 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.696281 30476 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.700984 28250 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.710650 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d939e5035541a4b003c5d276493d5c with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:08.711151 28791 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:08.712527 30476 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:00:08.712970 28726 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d939e5035541a4b003c5d276493d5c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:08.713111 2813 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:08.716792 28660 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240502 08:00:08.717289 3275 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.719019 3577 tablet_replica.cc:330] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:08.719298 3577 raft_consensus.cc:2229] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:08.719542 3577 raft_consensus.cc:2258] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:08.725790 28530 consensus_queue.cc:237] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:08.730165 28791 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:08.731285 991 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:08.732712 30476 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:08.734607 3245 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:08.735548 3186 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:08.736266 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.737259 28244 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 21d939e5035541a4b003c5d276493d5c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:08.738205 30476 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:08.739017 3577 ts_tablet_manager.cc:1875] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:08.741987 28250 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:08.743070 3577 ts_tablet_manager.cc:1888] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:08.743196 3577 log.cc:1196] T 21d939e5035541a4b003c5d276493d5c P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/21d939e5035541a4b003c5d276493d5c
I20240502 08:00:08.745002 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 21d939e5035541a4b003c5d276493d5c (table rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]) successfully deleted
I20240502 08:00:08.750599 28397 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:08.751176 991 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.763293 991 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.769898 28791 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.773115 28255 catalog_manager.cc:5484] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.774271 30477 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:08.777853 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet a7fd7f8583e14c91a932767ebe23b419 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:08.782800 3603 tablet_replica.cc:330] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:08.783061 3603 raft_consensus.cc:2229] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:08.783321 3603 raft_consensus.cc:2258] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:08.786886 3603 ts_tablet_manager.cc:1875] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:08.791476 3603 ts_tablet_manager.cc:1888] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:08.791667 3603 log.cc:1196] T a7fd7f8583e14c91a932767ebe23b419 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/a7fd7f8583e14c91a932767ebe23b419
I20240502 08:00:08.792330 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet a7fd7f8583e14c91a932767ebe23b419 (table rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]) successfully deleted
I20240502 08:00:08.841172 28529 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:08.845237 30215 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.845237 30345 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.845319 28791 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:08.845814 991 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.846032 3245 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.846378 3245 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:08.850440 30195 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.857129 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:08.861250 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:08.863029 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.864724 3581 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.870258 30215 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.874200 28639 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 08:00:08.874251 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:08.877646 28256 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.880124 28791 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.884047 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d1c1fa046b741aa8cf296aa57847fe3 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:08.892637 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:08.894225 30345 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:08.910167 3707 tablet_replica.cc:330] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:08.910400 3707 raft_consensus.cc:2229] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:08.910606 3707 raft_consensus.cc:2258] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:08.911202 3707 ts_tablet_manager.cc:1875] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:08.913770 3707 ts_tablet_manager.cc:1888] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:08.913872 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:08.913913 3707 log.cc:1196] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/5d1c1fa046b741aa8cf296aa57847fe3
I20240502 08:00:08.914448 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 5d1c1fa046b741aa8cf296aa57847fe3 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:08.961351 28661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef"
dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56230
I20240502 08:00:08.961503 28661 raft_consensus.cc:595] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:08.977012 1746 raft_consensus.cc:979] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: : Instructing follower 6c68c916f39a45f697df7eaf8973225a to start an election
I20240502 08:00:08.977108 1746 raft_consensus.cc:1067] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Signalling peer 6c68c916f39a45f697df7eaf8973225a to start an election
I20240502 08:00:08.978978 30215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef"
dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
from {username='slave'} at 127.16.36.195:35367
I20240502 08:00:08.979223 30215 raft_consensus.cc:482] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:08.979295 30215 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.980505 30215 raft_consensus.cc:504] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:08.982262 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 08:00:08.982417 28661 raft_consensus.cc:3032] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:08.982471 28661 raft_consensus.cc:729] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 08:00:08.982645 28661 consensus_queue.cc:260] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:08.982844 28661 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.983978 28661 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c68c916f39a45f697df7eaf8973225a in term 2.
I20240502 08:00:08.985296 30345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6efe17dcca7b4babad4785d19552b699"
I20240502 08:00:08.985496 30345 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.986778 30345 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c68c916f39a45f697df7eaf8973225a in term 2.
I20240502 08:00:08.987506 30150 leader_election.cc:302] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, 6efe17dcca7b4babad4785d19552b699, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 08:00:08.987823 3706 raft_consensus.cc:2781] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:08.987985 3706 raft_consensus.cc:686] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 2 LEADER]: Becoming Leader. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Running, Role: LEADER
I20240502 08:00:08.988314 3706 consensus_queue.cc:237] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:08.991727 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de90ee4ef90e4cb69ec37d7d9f0c86ef" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:08.991864 28529 raft_consensus.cc:3037] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:08.992954 28529 raft_consensus.cc:2454] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c68c916f39a45f697df7eaf8973225a in term 2.
I20240502 08:00:08.995208 30215 leader_election.cc:288] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 2 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 08:00:08.995522 28251 catalog_manager.cc:5484] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a reported cstate change: term changed from 1 to 2, leader changed from ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) to 6c68c916f39a45f697df7eaf8973225a (127.16.36.197). New cstate: current_term: 2 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:09.020524 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.031502 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.034613 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.051070 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.064658 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.096964 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.101850 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.105654 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.113020 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.132807 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.138600 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.149576 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.169554 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.181282 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.182435 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.192095 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.196192 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.208663 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.210160 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.229795 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.232960 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.237787 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.242676 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.247694 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.259196 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.262305 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.287858 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.297050 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.298226 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.315534 3897 ts_tablet_manager.cc:902] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:09.316531 3897 tablet_copy_client.cc:308] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:09.317171 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 21d939e5035541a4b003c5d276493d5c from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:39313)
I20240502 08:00:09.317291 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 21d939e5035541a4b003c5d276493d5c from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:39313: session id = 71922d97e1a4414698ecd14a7b2dedec-21d939e5035541a4b003c5d276493d5c
I20240502 08:00:09.318231 28812 tablet_copy_source_session.cc:205] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:09.318298 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.345278 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.346206 3897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:09.349805 3897 tablet_copy_client.cc:783] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:09.350157 3897 tablet_copy_client.cc:647] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:09.358120 3897 tablet_copy_client.cc:523] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:09.360860 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.361996 3897 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:00:09.393944 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.394675 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.397562 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.403976 30345 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:09.404632 3785 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:09.424396 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.429260 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.435344 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.465730 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.471087 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.484995 28661 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:09.486290 30214 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:00:09.486711 3219 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:09.490777 28529 consensus_queue.cc:237] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:09.492295 30345 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:09.492544 28790 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:09.492846 3693 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:09.493475 3693 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:09.495270 3693 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.495582 28790 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.496423 3897 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:09.497411 3897 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:00:09.497645 3897 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.136s user 0.030s sys 0.001s
I20240502 08:00:09.498283 3897 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:09.498396 3897 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:00:09.498354 28248 catalog_manager.cc:5484] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.498548 3897 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:09.498947 3897 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:09.499846 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-21d939e5035541a4b003c5d276493d5c received from {username='slave'} at 127.16.36.193:39313
I20240502 08:00:09.499928 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-21d939e5035541a4b003c5d276493d5c on tablet 21d939e5035541a4b003c5d276493d5c with peer 71922d97e1a4414698ecd14a7b2dedec
I20240502 08:00:09.501904 1746 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:09.502512 30215 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:09.505924 28253 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:09.506881 28397 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:09.508492 30477 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:09.508908 3960 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:09.510396 28790 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:09.510824 3960 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:09.512882 30477 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:09.513407 28529 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:09.513847 3785 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:09.514659 3960 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:09.515657 30345 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.522552 28256 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:09.523176 28791 consensus_queue.cc:237] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:09.524710 30345 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:09.525139 4006 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:09.525616 30215 raft_consensus.cc:1261] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:09.526296 3567 consensus_queue.cc:1035] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:09.528363 28661 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:09.528946 3785 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:09.529379 4014 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.530299 30215 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.530408 30345 raft_consensus.cc:2932] T 14bbfb3758e04aeab0df4bbab8380061 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:09.534081 28790 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:09.544235 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet ab785e07fa8248e5889bea8ac755d002 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:09.546083 28660 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:09.548175 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet a28d406b43194bc6b775707d021618f0 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:09.549011 30215 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:00:09.549083 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 698f494a22d34e659766b97cb24d0293 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:09.549456 1746 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:09.549860 28257 catalog_manager.cc:5484] T 14bbfb3758e04aeab0df4bbab8380061 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
W20240502 08:00:09.550346 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 698f494a22d34e659766b97cb24d0293. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:09.551822 4043 tablet_replica.cc:330] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:09.552338 4042 tablet_replica.cc:330] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:09.552608 4042 raft_consensus.cc:2229] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:09.552860 4042 raft_consensus.cc:2258] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:09.553332 3219 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:09.552780 4043 raft_consensus.cc:2229] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:09.553782 4043 raft_consensus.cc:2258] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:09.553838 4041 tablet_replica.cc:330] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:09.553963 4042 ts_tablet_manager.cc:1875] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:09.554356 4043 ts_tablet_manager.cc:1875] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:09.554409 4041 raft_consensus.cc:2229] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:09.554632 4041 raft_consensus.cc:2258] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:09.555020 28243 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 698f494a22d34e659766b97cb24d0293 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:09.555948 28251 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:09.556434 30215 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:09.557673 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 14bbfb3758e04aeab0df4bbab8380061 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:09.558195 4041 ts_tablet_manager.cc:1875] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:09.562109 4042 ts_tablet_manager.cc:1888] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:09.562325 4042 log.cc:1196] T a28d406b43194bc6b775707d021618f0 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/a28d406b43194bc6b775707d021618f0
I20240502 08:00:09.563023 4043 ts_tablet_manager.cc:1888] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:00:09.563541 4043 log.cc:1196] T 698f494a22d34e659766b97cb24d0293 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/698f494a22d34e659766b97cb24d0293
I20240502 08:00:09.564055 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet a28d406b43194bc6b775707d021618f0 (table rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]) successfully deleted
I20240502 08:00:09.564893 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 698f494a22d34e659766b97cb24d0293 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:09.565385 4042 tablet_replica.cc:330] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:09.565688 4041 ts_tablet_manager.cc:1888] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:09.565768 4041 log.cc:1196] T ab785e07fa8248e5889bea8ac755d002 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/ab785e07fa8248e5889bea8ac755d002
I20240502 08:00:09.566718 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet ab785e07fa8248e5889bea8ac755d002 (table rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]) successfully deleted
I20240502 08:00:09.567639 4042 raft_consensus.cc:2229] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:09.567890 4042 raft_consensus.cc:2258] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:09.568656 4042 ts_tablet_manager.cc:1875] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:09.571255 4042 ts_tablet_manager.cc:1888] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:00:09.571408 4042 log.cc:1196] T 14bbfb3758e04aeab0df4bbab8380061 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/14bbfb3758e04aeab0df4bbab8380061
I20240502 08:00:09.572685 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 14bbfb3758e04aeab0df4bbab8380061 (table rebalance_test_table_1 [id=1e2b1140681b44a284e5bcb21ee79110]) successfully deleted
I20240502 08:00:09.690448 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.710347 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.711899 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.729094 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.739415 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.748338 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.752331 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.757366 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.761471 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.776711 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.777154 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.777928 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.779981 28397 raft_consensus.cc:1203] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:00:09.784190 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.792307 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.795711 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.801606 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.809011 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:09.816126 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.819111 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.825240 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.847730 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.849042 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:09.881881 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:09.889587 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.989198 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:09.991286 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:09.999372 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.011859 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:10.024385 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.032724 30606 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:10.034250 28529 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 08:00:10.034686 3595 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.035580 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:10.035867 28660 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 3 index: 7. (index mismatch)
I20240502 08:00:10.037185 3595 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.037751 3595 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 3 LEADER]: Committing config change with OpId 3.7: config changed from index 4 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.041371 28529 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 4 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.047413 28255 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 reported cstate change: config changed from index 4 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 3 leader_uuid: "e5e9553dabc94f218f6427c1685ffc38" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.051342 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet c65fe7f823df4b4996769fc784d3adb7 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:10.051632 4041 tablet_replica.cc:330] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:10.051754 4041 raft_consensus.cc:2229] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:10.051858 4041 raft_consensus.cc:2258] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:10.052600 4041 ts_tablet_manager.cc:1875] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:10.055136 4041 ts_tablet_manager.cc:1888] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.6
I20240502 08:00:10.055223 4041 log.cc:1196] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/c65fe7f823df4b4996769fc784d3adb7
I20240502 08:00:10.057168 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:10.057220 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet c65fe7f823df4b4996769fc784d3adb7 (table rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]) successfully deleted
I20240502 08:00:10.058490 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.063771 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:10.067755 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:10.086799 28660 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Committing config change with OpId 3.7: config changed from index 4 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.088599 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.090474 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.097528 4333 ts_tablet_manager.cc:902] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:10.104780 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.113001 4333 tablet_copy_client.cc:308] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:10.113480 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 698f494a22d34e659766b97cb24d0293 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:43285)
I20240502 08:00:10.113593 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 698f494a22d34e659766b97cb24d0293 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:43285: session id = e5e9553dabc94f218f6427c1685ffc38-698f494a22d34e659766b97cb24d0293
I20240502 08:00:10.114504 28681 tablet_copy_source_session.cc:205] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:10.115794 4333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:10.118772 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.128805 28660 consensus_queue.cc:237] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:10.130455 30215 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:10.131155 28790 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:00:10.131875 1746 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:10.132335 1746 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:00:10.138129 28660 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:10.141005 28397 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:10.141615 30477 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:10.142345 3219 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:10.142757 3219 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:10.144518 30606 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:10.144958 4333 tablet_copy_client.cc:783] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:10.144975 1746 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:10.145120 4333 tablet_copy_client.cc:647] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:10.147341 4333 tablet_copy_client.cc:523] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:10.147632 1746 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.148898 4333 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:10.150365 28250 catalog_manager.cc:5484] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.150693 28790 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.153905 30214 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.154341 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:10.154580 4041 tablet_replica.cc:330] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:10.154747 4041 raft_consensus.cc:2229] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:10.154917 4041 raft_consensus.cc:2258] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:10.154915 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.155436 4041 ts_tablet_manager.cc:1875] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:10.155984 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.157855 4041 ts_tablet_manager.cc:1888] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 08:00:10.157940 4041 log.cc:1196] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ba62cb7ab7b429d85e0ac037c127cb4
I20240502 08:00:10.158461 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 2ba62cb7ab7b429d85e0ac037c127cb4 (table rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]) successfully deleted
I20240502 08:00:10.163794 1746 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.170528 28397 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.171106 30606 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.173168 28255 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.174062 30477 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.176541 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb29b8b566747e5b82a8969a070123e with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:10.195772 4333 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:10.196144 4333 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:10.196331 4333 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.047s user 0.023s sys 0.001s
I20240502 08:00:10.196830 4333 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:10.196944 4333 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:10.197060 4333 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:10.197324 4333 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 08:00:10.198601 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-698f494a22d34e659766b97cb24d0293 received from {username='slave'} at 127.16.36.200:43285
I20240502 08:00:10.198691 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-698f494a22d34e659766b97cb24d0293 on tablet 698f494a22d34e659766b97cb24d0293 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:10.208156 4385 tablet_replica.cc:330] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:10.210036 4385 raft_consensus.cc:2229] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:10.210392 4385 raft_consensus.cc:2258] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:10.211093 4385 ts_tablet_manager.cc:1875] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:10.216591 4385 ts_tablet_manager.cc:1888] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:10.216786 4385 log.cc:1196] T 4cb29b8b566747e5b82a8969a070123e P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/4cb29b8b566747e5b82a8969a070123e
I20240502 08:00:10.217530 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 4cb29b8b566747e5b82a8969a070123e (table rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]) successfully deleted
I20240502 08:00:10.235366 28529 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3"
dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36382
I20240502 08:00:10.235535 28529 raft_consensus.cc:595] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:10.238512 28790 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:10.240077 30606 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:10.240092 30477 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:10.240603 4006 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.240931 4006 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.244817 4006 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.246920 30477 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.250052 28250 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.253774 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a40382745f84f0eadf9e0ee2e057289 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:10.254022 4385 tablet_replica.cc:330] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:10.254173 4385 raft_consensus.cc:2229] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:10.254354 4385 raft_consensus.cc:2258] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:10.254292 30606 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:10.254978 4385 ts_tablet_manager.cc:1875] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:10.258510 4385 ts_tablet_manager.cc:1888] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:00:10.258761 4385 log.cc:1196] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a40382745f84f0eadf9e0ee2e057289
I20240502 08:00:10.259636 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 6a40382745f84f0eadf9e0ee2e057289 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:10.339303 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:10.344889 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:10.349097 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:10.353152 4422 raft_consensus.cc:979] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: : Instructing follower de423605477a4819a9569821f59f03e4 to start an election
I20240502 08:00:10.353250 4422 raft_consensus.cc:1067] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Signalling peer de423605477a4819a9569821f59f03e4 to start an election
I20240502 08:00:10.353654 28790 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3"
dest_uuid: "de423605477a4819a9569821f59f03e4"
from {username='slave'} at 127.16.36.194:51335
I20240502 08:00:10.353796 28790 raft_consensus.cc:482] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:10.353852 28790 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.354961 28790 raft_consensus.cc:504] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:10.355453 28790 leader_election.cc:288] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 2 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:10.356030 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:10.356185 28529 raft_consensus.cc:3032] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:10.356238 28529 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:10.356382 28529 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:10.356628 28529 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.357667 30214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:10.357731 28529 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:10.357795 30214 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.358422 30345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "de423605477a4819a9569821f59f03e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6efe17dcca7b4babad4785d19552b699"
I20240502 08:00:10.358558 30345 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.358873 30214 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:10.359582 30345 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de423605477a4819a9569821f59f03e4 in term 2.
I20240502 08:00:10.360158 28725 leader_election.cc:302] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:10.360483 4006 raft_consensus.cc:2781] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:10.360567 4006 raft_consensus.cc:686] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Becoming Leader. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 08:00:10.360702 4006 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:10.361138 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.366297 28255 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 reported cstate change: term changed from 1 to 2, leader changed from c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) to de423605477a4819a9569821f59f03e4 (127.16.36.196). New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.374512 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.408952 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.409144 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.414083 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:10.417546 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.429899 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.432672 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.457553 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.458407 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.463374 30457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:00:10.477452 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.478843 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.481725 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.494221 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.494690 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.497143 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:10.500026 28378 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:10.523437 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.531962 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.546684 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.572566 28397 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319"
dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39538
I20240502 08:00:10.572736 28397 raft_consensus.cc:595] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:10.607865 4348 raft_consensus.cc:979] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: : Instructing follower 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:10.607964 4348 raft_consensus.cc:1067] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Signalling peer 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:10.608520 30477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319"
dest_uuid: "29126d5aee924f81bdd47cc7153420a1"
from {username='slave'} at 127.16.36.193:46783
I20240502 08:00:10.608723 30477 raft_consensus.cc:482] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:10.608785 30477 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.609779 30477 raft_consensus.cc:504] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:10.611140 30477 leader_election.cc:288] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 2 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 08:00:10.611879 28790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 08:00:10.612013 28790 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.612344 28529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:10.612604 28529 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.613106 28790 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:10.613480 28529 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:10.613813 30410 leader_election.cc:302] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 29126d5aee924f81bdd47cc7153420a1, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:10.614326 4384 raft_consensus.cc:2781] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:10.614512 4384 raft_consensus.cc:686] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Becoming Leader. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Running, Role: LEADER
I20240502 08:00:10.614869 4384 consensus_queue.cc:237] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:10.617416 28397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 08:00:10.617551 28397 raft_consensus.cc:3032] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:10.617609 28397 raft_consensus.cc:729] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 08:00:10.617740 28397 consensus_queue.cc:260] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:10.617919 28397 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.617799 28255 catalog_manager.cc:5484] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: term changed from 1 to 2, leader changed from 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) to 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199). New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:10.618808 28397 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:10.653445 30345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1631d3e38209497ea2454fdfc555a319" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6efe17dcca7b4babad4785d19552b699"
I20240502 08:00:10.653605 30345 raft_consensus.cc:3037] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:10.654755 30345 raft_consensus.cc:2454] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:10.660050 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.677618 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.678129 30606 raft_consensus.cc:1203] T 698f494a22d34e659766b97cb24d0293 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:00:10.699761 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.701066 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.703147 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.708993 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.710675 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:10.717808 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.722947 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.727556 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.739924 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.794603 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.797696 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.804934 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.805239 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.808830 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.809228 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:10.825229 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:10.836851 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.838229 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.852733 28529 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:10.854382 4006 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.856640 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.859961 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:10.861997 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:10.880244 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:10.922423 30345 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:10.922966 4006 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.932600 30214 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:10.933267 4444 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.948504 28790 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:10.950114 30214 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:10.950575 4812 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:10.955662 4812 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.956975 30214 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.960326 28252 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:10.964768 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:10.964887 28790 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
W20240502 08:00:10.966570 28727 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0499a3f3e6bd4069bf992e32f1abfdb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:10.967669 30214 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:10.968122 4812 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:10.969044 4861 tablet_replica.cc:330] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:10.969226 4861 raft_consensus.cc:2229] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:10.969374 4861 raft_consensus.cc:2258] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:10.970199 4861 ts_tablet_manager.cc:1875] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:10.970789 4444 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:10.971303 30214 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:10.973232 28244 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:10.973657 4861 ts_tablet_manager.cc:1888] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:00:10.973731 4861 log.cc:1196] T 0499a3f3e6bd4069bf992e32f1abfdb0 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/0499a3f3e6bd4069bf992e32f1abfdb0
I20240502 08:00:10.974262 28252 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 7 to 8, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:10.974906 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 0499a3f3e6bd4069bf992e32f1abfdb0 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:10.976581 28397 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:10.978195 30477 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:10.978632 4000 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:10.983774 4000 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.986742 28255 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:10.986716 30477 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:11.000298 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet a28d406b43194bc6b775707d021618f0 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:11.001104 28397 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.003540 30477 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:11.003845 4905 tablet_replica.cc:330] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:11.004025 4905 raft_consensus.cc:2229] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:11.004209 4905 raft_consensus.cc:2258] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:11.004863 4905 ts_tablet_manager.cc:1875] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:11.005065 4000 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:11.007495 4905 ts_tablet_manager.cc:1888] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:11.007580 4905 log.cc:1196] T a28d406b43194bc6b775707d021618f0 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/a28d406b43194bc6b775707d021618f0
I20240502 08:00:11.008145 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet a28d406b43194bc6b775707d021618f0 (table rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]) successfully deleted
W20240502 08:00:11.010183 28335 consensus_peers.cc:485] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a28d406b43194bc6b775707d021618f0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:11.013208 4000 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.015384 28242 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a28d406b43194bc6b775707d021618f0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:11.015535 30477 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.016387 28256 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 6 to 7, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.021025 30345 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:11.021589 4666 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:11.047490 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.047804 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.063082 28528 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:11.063588 4666 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:11.064028 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.068917 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.069841 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.101536 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.103008 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.113130 28790 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:11.113593 4384 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:11.114300 28397 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:00:11.115386 4666 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:11.118846 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.125077 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.143261 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.153049 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.156438 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.160743 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.182924 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.192870 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.209414 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.253119 28790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3"
dest_uuid: "de423605477a4819a9569821f59f03e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56664
I20240502 08:00:11.253340 28790 raft_consensus.cc:595] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Received request to transfer leadership
I20240502 08:00:11.267027 28790 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3"
dest_uuid: "de423605477a4819a9569821f59f03e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56676
I20240502 08:00:11.267202 28790 raft_consensus.cc:595] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Received request to transfer leadership
I20240502 08:00:11.291729 30195 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.302908 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.319584 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.323058 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.330199 28771 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.332237 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.333851 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.334800 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.355026 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.356377 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.367179 4812 raft_consensus.cc:979] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: : Instructing follower 6c68c916f39a45f697df7eaf8973225a to start an election
I20240502 08:00:11.367301 4812 raft_consensus.cc:1067] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Signalling peer 6c68c916f39a45f697df7eaf8973225a to start an election
I20240502 08:00:11.368314 30215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3"
dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
from {username='slave'} at 127.16.36.196:50907
I20240502 08:00:11.368497 30215 raft_consensus.cc:482] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:11.368582 30215 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:11.369796 30215 raft_consensus.cc:504] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.370330 30215 leader_election.cc:288] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 3 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 08:00:11.370635 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:11.370779 28528 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:11.371702 28790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 08:00:11.371861 28790 raft_consensus.cc:3032] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:00:11.371907 28790 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 08:00:11.371906 28528 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c68c916f39a45f697df7eaf8973225a in term 3.
I20240502 08:00:11.372093 28790 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.372285 28790 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:11.373452 30345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d1c1fa046b741aa8cf296aa57847fe3" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: true dest_uuid: "6efe17dcca7b4babad4785d19552b699"
I20240502 08:00:11.373657 30345 raft_consensus.cc:3037] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:11.374241 28790 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c68c916f39a45f697df7eaf8973225a in term 3.
I20240502 08:00:11.374415 30345 raft_consensus.cc:2454] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c68c916f39a45f697df7eaf8973225a in term 3.
I20240502 08:00:11.374680 30151 leader_election.cc:302] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:11.374902 4332 raft_consensus.cc:2781] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:00:11.375015 4332 raft_consensus.cc:686] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 LEADER]: Becoming Leader. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Running, Role: LEADER
I20240502 08:00:11.375164 4332 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.376963 28250 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: term changed from 2 to 3, leader changed from de423605477a4819a9569821f59f03e4 (127.16.36.196) to 6c68c916f39a45f697df7eaf8973225a (127.16.36.197). New cstate: current_term: 3 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.380127 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.402534 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:11.403383 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.403699 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.404232 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.426959 5110 ts_tablet_manager.cc:902] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:11.427732 5110 tablet_copy_client.cc:308] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:11.428881 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet a28d406b43194bc6b775707d021618f0 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:50683)
I20240502 08:00:11.429018 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet a28d406b43194bc6b775707d021618f0 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:50683: session id = 6c68c916f39a45f697df7eaf8973225a-a28d406b43194bc6b775707d021618f0
I20240502 08:00:11.430028 28419 tablet_copy_source_session.cc:205] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:11.435055 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28d406b43194bc6b775707d021618f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:11.439713 5110 tablet_copy_client.cc:783] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:11.440542 5110 tablet_copy_client.cc:647] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:11.447829 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.455739 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.456560 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:11.464071 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.465209 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:11.476207 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.488512 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.489286 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.499869 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:11.545202 5110 tablet_copy_client.cc:523] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:11.546700 5110 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:11.572685 5153 ts_tablet_manager.cc:902] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:11.573392 5153 tablet_copy_client.cc:308] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:11.574060 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:47747)
I20240502 08:00:11.574198 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 0499a3f3e6bd4069bf992e32f1abfdb0 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:47747: session id = 6efe17dcca7b4babad4785d19552b699-0499a3f3e6bd4069bf992e32f1abfdb0
I20240502 08:00:11.575232 28812 tablet_copy_source_session.cc:205] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:11.585630 5153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0499a3f3e6bd4069bf992e32f1abfdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:11.590335 5153 tablet_copy_client.cc:783] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:11.590530 5153 tablet_copy_client.cc:647] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:11.594017 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.595597 5153 tablet_copy_client.cc:523] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:11.597750 5153 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:11.612911 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.622020 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:11.629945 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.632349 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.633147 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.636379 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:11.649518 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:11.650230 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.653707 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.683349 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.691694 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.700106 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.707175 5110 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:11.707507 5110 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:11.707690 5110 ts_tablet_manager.cc:1367] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.161s user 0.029s sys 0.000s
I20240502 08:00:11.707731 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.708127 5110 raft_consensus.cc:348] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.708241 5110 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:11.709255 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:11.712440 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.726919 5110 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.727308 5110 ts_tablet_manager.cc:1398] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240502 08:00:11.727842 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-a28d406b43194bc6b775707d021618f0 received from {username='slave'} at 127.16.36.197:50683
I20240502 08:00:11.727921 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-a28d406b43194bc6b775707d021618f0 on tablet a28d406b43194bc6b775707d021618f0 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:11.731184 30606 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 3.7, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:11.732419 28528 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 3 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 08:00:11.733060 4325 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:11.745075 4325 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 3 LEADER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:11.747692 28528 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Committing config change with OpId 3.8: config changed from index 7 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:11.747875 28252 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 reported cstate change: config changed from index 7 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 3 leader_uuid: "e5e9553dabc94f218f6427c1685ffc38" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:11.750607 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet c65fe7f823df4b4996769fc784d3adb7 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:11.752419 30606 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.756664 28528 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Refusing update from remote peer e5e9553dabc94f218f6427c1685ffc38: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 08:00:11.757133 4325 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:11.760743 4325 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.762267 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c65fe7f823df4b4996769fc784d3adb7 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:11.763010 30345 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 08:00:11.763000 28528 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 8 to 9, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.763558 4332 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:11.767747 5241 tablet_replica.cc:330] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:11.768749 5241 raft_consensus.cc:2229] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:11.768687 28250 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 reported cstate change: config changed from index 8 to 9, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 3 leader_uuid: "e5e9553dabc94f218f6427c1685ffc38" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.769475 5241 raft_consensus.cc:2258] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:11.771521 5241 ts_tablet_manager.cc:1875] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:11.785594 5241 ts_tablet_manager.cc:1888] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.7
W20240502 08:00:11.788316 30544 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c65fe7f823df4b4996769fc784d3adb7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:11.792702 5241 log.cc:1196] T c65fe7f823df4b4996769fc784d3adb7 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/c65fe7f823df4b4996769fc784d3adb7
I20240502 08:00:11.795408 28397 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:11.795984 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet c65fe7f823df4b4996769fc784d3adb7 (table rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]) successfully deleted
I20240502 08:00:11.797649 30606 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:11.798134 4916 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:11.800679 5263 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:11.801049 30606 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:11.803560 28252 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 5 to 6, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:11.806898 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 3598dbda342f49aea2384b5beb17d7a1 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:11.807850 28397 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:11.809839 30606 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:00:11.810107 28334 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): Couldn't send request to peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3598dbda342f49aea2384b5beb17d7a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:11.810252 4916 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:11.812652 5153 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:11.812901 5153 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:11.813076 5153 ts_tablet_manager.cc:1367] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.215s user 0.030s sys 0.000s
I20240502 08:00:11.813563 5153 raft_consensus.cc:348] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:11.813683 5153 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:11.813903 5153 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:11.814652 5153 ts_tablet_manager.cc:1398] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:00:11.815101 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-0499a3f3e6bd4069bf992e32f1abfdb0 received from {username='slave'} at 127.16.36.198:47747
I20240502 08:00:11.815166 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-0499a3f3e6bd4069bf992e32f1abfdb0 on tablet 0499a3f3e6bd4069bf992e32f1abfdb0 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:11.817664 4916 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:11.818495 5268 tablet_replica.cc:330] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:11.818703 5268 raft_consensus.cc:2229] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:11.818894 5268 raft_consensus.cc:2258] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:11.819559 5268 ts_tablet_manager.cc:1875] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:11.820034 28242 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3598dbda342f49aea2384b5beb17d7a1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:11.820709 30606 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:11.821163 28250 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 6 to 7, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:11.822659 5268 ts_tablet_manager.cc:1888] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:11.824347 5268 log.cc:1196] T 3598dbda342f49aea2384b5beb17d7a1 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/3598dbda342f49aea2384b5beb17d7a1
I20240502 08:00:11.825466 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 3598dbda342f49aea2384b5beb17d7a1 (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
I20240502 08:00:11.829326 28660 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:11.830673 30214 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:11.831578 30345 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:11.832118 4361 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:11.832759 4361 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:11.835870 4361 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.838912 30345 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.840128 28250 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.843469 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet afc9c4d9dd5b4c039686ccd918616859 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:11.843736 5268 tablet_replica.cc:330] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:11.843869 5268 raft_consensus.cc:2229] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:11.844026 5268 raft_consensus.cc:2258] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:11.844702 5268 ts_tablet_manager.cc:1875] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:11.844621 30214 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:11.847157 5268 ts_tablet_manager.cc:1888] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:00:11.847232 5268 log.cc:1196] T afc9c4d9dd5b4c039686ccd918616859 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/afc9c4d9dd5b4c039686ccd918616859
I20240502 08:00:11.848753 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet afc9c4d9dd5b4c039686ccd918616859 (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
I20240502 08:00:11.892855 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.894280 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:11.897290 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:11.899590 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.917240 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:11.933733 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:11.942992 30215 raft_consensus.cc:1203] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:00:11.945964 28791 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 08:00:11.948210 4332 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:11.952075 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:11.952155 30345 raft_consensus.cc:1203] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:00:11.960179 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:11.962586 28528 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20240502 08:00:11.963066 4332 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:12.029237 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.035740 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.039322 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.047613 28791 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0"
dest_uuid: "de423605477a4819a9569821f59f03e4"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55474
I20240502 08:00:12.047762 28791 raft_consensus.cc:595] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Received request to transfer leadership
I20240502 08:00:12.052891 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.055006 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:12.082157 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:12.085474 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.096109 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.127725 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.131996 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.140944 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.161486 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.161854 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.163779 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:12.164098 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:12.171470 5110 ts_tablet_manager.cc:902] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
I20240502 08:00:12.172051 5110 tablet_copy_client.cc:308] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.200:43189
I20240502 08:00:12.191805 30627 tablet_copy_service.cc:140] P e5e9553dabc94f218f6427c1685ffc38: Received BeginTabletCopySession request for tablet c65fe7f823df4b4996769fc784d3adb7 from peer 6c68c916f39a45f697df7eaf8973225a ({username='slave'} at 127.16.36.197:49337)
I20240502 08:00:12.191980 30627 tablet_copy_service.cc:161] P e5e9553dabc94f218f6427c1685ffc38: Beginning new tablet copy session on tablet c65fe7f823df4b4996769fc784d3adb7 from peer 6c68c916f39a45f697df7eaf8973225a at {username='slave'} at 127.16.36.197:49337: session id = 6c68c916f39a45f697df7eaf8973225a-c65fe7f823df4b4996769fc784d3adb7
I20240502 08:00:12.193462 30627 tablet_copy_source_session.cc:205] T c65fe7f823df4b4996769fc784d3adb7 P e5e9553dabc94f218f6427c1685ffc38: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:12.194905 5110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65fe7f823df4b4996769fc784d3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:12.197613 5110 tablet_copy_client.cc:783] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:12.197769 5110 tablet_copy_client.cc:647] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:12.213725 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.224613 5110 tablet_copy_client.cc:523] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:12.226328 5110 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap starting.
I20240502 08:00:12.236939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.237586 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.246037 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.252775 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.253746 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:12.264214 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.278116 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.301815 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:12.334319 28529 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:12.336293 30477 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:12.336895 5379 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:12.338590 5110 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:12.338907 5110 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: Bootstrap complete.
I20240502 08:00:12.339115 5110 ts_tablet_manager.cc:1367] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: Time spent bootstrapping tablet: real 0.113s user 0.029s sys 0.000s
I20240502 08:00:12.339552 5379 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:12.339649 5110 raft_consensus.cc:348] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:12.339763 5110 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68c916f39a45f697df7eaf8973225a, State: Initialized, Role: FOLLOWER
I20240502 08:00:12.339888 5110 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:12.340255 5110 ts_tablet_manager.cc:1398] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:12.341720 30627 tablet_copy_service.cc:342] P e5e9553dabc94f218f6427c1685ffc38: Request end of tablet copy session 6c68c916f39a45f697df7eaf8973225a-c65fe7f823df4b4996769fc784d3adb7 received from {username='slave'} at 127.16.36.197:49337
I20240502 08:00:12.341828 30627 tablet_copy_service.cc:433] P e5e9553dabc94f218f6427c1685ffc38: ending tablet copy session 6c68c916f39a45f697df7eaf8973225a-c65fe7f823df4b4996769fc784d3adb7 on tablet c65fe7f823df4b4996769fc784d3adb7 with peer 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:12.342780 30477 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:12.342953 28255 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:12.353221 28397 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:12.355147 28791 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:12.356339 5263 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.357620 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 7dbf7921ae84476ab53fbd4707f463eb with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:12.358278 28529 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:12.359489 5617 tablet_replica.cc:330] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:12.360095 5617 raft_consensus.cc:2229] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:12.360365 5617 raft_consensus.cc:2258] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:12.361100 5617 ts_tablet_manager.cc:1875] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:12.361613 30477 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:00:12.362372 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbf7921ae84476ab53fbd4707f463eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:12.362594 5378 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.364382 5617 ts_tablet_manager.cc:1888] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:00:12.364550 5617 log.cc:1196] T 7dbf7921ae84476ab53fbd4707f463eb P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/7dbf7921ae84476ab53fbd4707f463eb
I20240502 08:00:12.365218 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 7dbf7921ae84476ab53fbd4707f463eb (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
I20240502 08:00:12.365689 5378 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.367632 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7dbf7921ae84476ab53fbd4707f463eb with cas_config_opid_index 6: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:12.368613 30477 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.371651 5263 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:12.372150 28257 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.373059 28791 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:12.379297 28248 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 6 to 7, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:12.384162 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet 57f7c358ace742649df52380939ece9d with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:12.384510 5617 tablet_replica.cc:330] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:12.384678 5617 raft_consensus.cc:2229] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:12.384846 5617 raft_consensus.cc:2258] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:12.385447 5617 ts_tablet_manager.cc:1875] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:12.388366 28397 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
W20240502 08:00:12.391705 28332 consensus_peers.cc:485] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec -> Peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): Couldn't send request to peer c39bcb719db947b8afb0bb141e1f7a4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57f7c358ace742649df52380939ece9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:12.392366 28791 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:12.392961 5637 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:12.394347 5617 ts_tablet_manager.cc:1888] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:00:12.394556 5617 log.cc:1196] T 57f7c358ace742649df52380939ece9d P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/57f7c358ace742649df52380939ece9d
I20240502 08:00:12.395259 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 57f7c358ace742649df52380939ece9d (table rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]) successfully deleted
I20240502 08:00:12.395898 5637 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:12.396296 28791 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:12.399430 28253 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:12.400492 28242 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 57f7c358ace742649df52380939ece9d with cas_config_opid_index 7: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:12.403434 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.414570 28771 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:00:12.416720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.421490 30457 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.425711 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.433646 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.433982 5389 raft_consensus.cc:979] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: : Instructing follower 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:12.434090 5389 raft_consensus.cc:1067] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Signalling peer 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:12.434545 30477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0"
dest_uuid: "29126d5aee924f81bdd47cc7153420a1"
from {username='slave'} at 127.16.36.196:60891
I20240502 08:00:12.434701 30477 raft_consensus.cc:482] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:12.434773 30477 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:12.438422 30477 raft_consensus.cc:504] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:12.439950 30477 leader_election.cc:288] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 3 election: Requested vote from peers de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:12.441258 28791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 08:00:12.441390 28791 raft_consensus.cc:3032] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 08:00:12.441426 28791 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Running, Role: LEADER
I20240502 08:00:12.441587 28791 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:12.441745 28791 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:12.442858 28791 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 3.
I20240502 08:00:12.443246 30413 leader_election.cc:302] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29126d5aee924f81bdd47cc7153420a1, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:12.443599 4976 raft_consensus.cc:2781] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 08:00:12.443786 4976 raft_consensus.cc:686] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 3 LEADER]: Becoming Leader. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Running, Role: LEADER
I20240502 08:00:12.443951 4976 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:12.446128 28248 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: term changed from 2 to 3, leader changed from de423605477a4819a9569821f59f03e4 (127.16.36.196) to 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199). New cstate: current_term: 3 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:12.449563 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.457348 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.471547 5662 ts_tablet_manager.cc:902] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:12.472380 5662 tablet_copy_client.cc:308] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:12.472997 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 3598dbda342f49aea2384b5beb17d7a1 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 ({username='slave'} at 127.16.36.195:55199)
I20240502 08:00:12.473055 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:12.473134 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 3598dbda342f49aea2384b5beb17d7a1 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 at {username='slave'} at 127.16.36.195:55199: session id = ed3ca7e829bc42b5a8e9a7d1bdcc5a44-3598dbda342f49aea2384b5beb17d7a1
I20240502 08:00:12.474048 28419 tablet_copy_source_session.cc:205] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:12.475284 5662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3598dbda342f49aea2384b5beb17d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:12.480857 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04d358245cdf4b2ebb0b5005f75cf1e0" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: true dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:12.481006 30213 raft_consensus.cc:3037] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Advancing to term 3
I20240502 08:00:12.482060 30213 raft_consensus.cc:2454] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 3.
I20240502 08:00:12.487444 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:12.491909 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.501305 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:12.504529 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.508651 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.513182 5662 tablet_copy_client.cc:783] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:12.513846 5662 tablet_copy_client.cc:647] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:12.517294 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.520988 5662 tablet_copy_client.cc:523] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:12.525171 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.528225 5662 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 08:00:12.530512 30477 consensus_queue.cc:237] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:12.532270 30345 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:12.532853 4976 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.532912 28529 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:12.533327 4976 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.537103 28790 raft_consensus.cc:1261] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:12.540405 5665 consensus_queue.cc:1035] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.553289 30213 raft_consensus.cc:1203] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.8->[3.9-3.9] Dedup: 3.9->[]
I20240502 08:00:12.569921 5665 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.570272 28790 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.572947 30345 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.573278 28248 catalog_manager.cc:5484] T 1631d3e38209497ea2454fdfc555a319 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 5 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.575856 28529 raft_consensus.cc:2932] T 1631d3e38209497ea2454fdfc555a319 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.576396 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 1631d3e38209497ea2454fdfc555a319 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:12.588706 5700 tablet_replica.cc:330] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:12.588981 5700 raft_consensus.cc:2229] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:12.589190 5700 raft_consensus.cc:2258] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:12.591800 5700 ts_tablet_manager.cc:1875] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:12.595156 5700 ts_tablet_manager.cc:1888] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:00:12.595242 5700 log.cc:1196] T 1631d3e38209497ea2454fdfc555a319 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/1631d3e38209497ea2454fdfc555a319
I20240502 08:00:12.600752 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 1631d3e38209497ea2454fdfc555a319 (table rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]) successfully deleted
I20240502 08:00:12.622268 5662 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:12.622650 5662 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap complete.
I20240502 08:00:12.622853 5662 ts_tablet_manager.cc:1367] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.095s user 0.031s sys 0.000s
I20240502 08:00:12.623382 5662 raft_consensus.cc:348] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:12.623507 5662 raft_consensus.cc:729] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 08:00:12.623672 5662 consensus_queue.cc:260] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:12.624083 5662 ts_tablet_manager.cc:1398] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:12.624992 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-3598dbda342f49aea2384b5beb17d7a1 received from {username='slave'} at 127.16.36.195:55199
I20240502 08:00:12.625083 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-3598dbda342f49aea2384b5beb17d7a1 on tablet 3598dbda342f49aea2384b5beb17d7a1 with peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:00:12.672736 28790 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:12.675330 30213 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:12.676141 5389 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:12.677052 30345 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
W20240502 08:00:12.677546 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0499a3f3e6bd4069bf992e32f1abfdb0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:12.678725 5389 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:12.681268 5389 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.682300 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.684986 30213 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.685622 30345 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.687091 28257 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:12.694890 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.705003 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.708742 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.714454 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.725730 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:12.730669 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:12.773489 28529 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:12.775256 30606 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:12.775300 28397 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
W20240502 08:00:12.775437 28465 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3288b438832e488e8257227cb8934565. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:12.775697 3943 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.776048 3943 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:12.778339 5378 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.778667 28397 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.782816 28250 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 5 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.783037 30606 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:12.791203 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.823657 5903 ts_tablet_manager.cc:902] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:12.824618 5903 tablet_copy_client.cc:308] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:12.825333 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 57f7c358ace742649df52380939ece9d from peer c39bcb719db947b8afb0bb141e1f7a4e ({username='slave'} at 127.16.36.194:43003)
I20240502 08:00:12.825448 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 57f7c358ace742649df52380939ece9d from peer c39bcb719db947b8afb0bb141e1f7a4e at {username='slave'} at 127.16.36.194:43003: session id = c39bcb719db947b8afb0bb141e1f7a4e-57f7c358ace742649df52380939ece9d
I20240502 08:00:12.826453 28419 tablet_copy_source_session.cc:205] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:12.828646 5903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f7c358ace742649df52380939ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:12.831938 5903 tablet_copy_client.cc:783] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:12.832317 5903 tablet_copy_client.cc:647] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:12.834952 5903 tablet_copy_client.cc:523] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:12.837805 5903 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 08:00:12.844302 28660 raft_consensus.cc:1203] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:00:12.871378 28660 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:12.874580 30213 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:00:12.875128 4361 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:12.875699 30345 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:00:12.876261 4361 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:00:12.876638 28597 consensus_peers.cc:485] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): Couldn't send request to peer de423605477a4819a9569821f59f03e4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:12.881773 4361 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:12.885372 28257 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:12.887187 30213 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:12.890105 30345 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:12.898797 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.903044 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.911875 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.925617 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.925786 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.930191 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:12.934221 30213 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 3.8, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:12.936183 30345 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 08:00:12.937136 5393 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:12.938541 28790 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 08:00:12.939281 5393 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:12.939762 5393 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 LEADER]: Committing config change with OpId 3.9: config changed from index 6 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:12.942741 30345 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 6 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:12.943174 28253 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 6 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 3 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:12.947404 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d1c1fa046b741aa8cf296aa57847fe3 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:12.953537 28790 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Committing config change with OpId 3.9: config changed from index 6 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:12.957401 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.958911 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:12.960660 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:12.967370 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:12.967952 5903 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:12.968334 5903 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap complete.
I20240502 08:00:12.968575 5903 ts_tablet_manager.cc:1367] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.131s user 0.030s sys 0.002s
I20240502 08:00:12.969054 5903 raft_consensus.cc:348] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:12.969190 5903 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 08:00:12.969360 5903 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:12.970252 5903 ts_tablet_manager.cc:1398] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:00:12.981796 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-57f7c358ace742649df52380939ece9d received from {username='slave'} at 127.16.36.194:43003
I20240502 08:00:12.981892 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-57f7c358ace742649df52380939ece9d on tablet 57f7c358ace742649df52380939ece9d with peer c39bcb719db947b8afb0bb141e1f7a4e
I20240502 08:00:12.991171 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:12.992502 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:12.995662 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:13.007555 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.009258 6002 tablet_replica.cc:330] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:13.009526 6008 ts_tablet_manager.cc:902] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:13.010071 6008 tablet_copy_client.cc:308] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:13.010608 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 7dbf7921ae84476ab53fbd4707f463eb from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:38023)
I20240502 08:00:13.010738 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 7dbf7921ae84476ab53fbd4707f463eb from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:38023: session id = e5e9553dabc94f218f6427c1685ffc38-7dbf7921ae84476ab53fbd4707f463eb
I20240502 08:00:13.011503 28790 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 08:00:13.011734 28548 tablet_copy_source_session.cc:205] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.012012 5693 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:13.012609 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.012650 6002 raft_consensus.cc:2229] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:13.012879 6002 raft_consensus.cc:2258] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:13.014850 6002 ts_tablet_manager.cc:1875] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:13.017835 6002 ts_tablet_manager.cc:1888] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.8
I20240502 08:00:13.027858 6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf7921ae84476ab53fbd4707f463eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.036300 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.040704 30213 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240502 08:00:13.041239 5693 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:13.046234 6008 tablet_copy_client.cc:783] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.046483 6008 tablet_copy_client.cc:647] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.046847 6002 log.cc:1196] T 5d1c1fa046b741aa8cf296aa57847fe3 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/5d1c1fa046b741aa8cf296aa57847fe3
I20240502 08:00:13.047780 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet 5d1c1fa046b741aa8cf296aa57847fe3 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:13.048004 6008 tablet_copy_client.cc:523] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.049856 6008 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:13.089359 6008 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.089705 6008 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:13.089924 6008 ts_tablet_manager.cc:1367] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.000s
I20240502 08:00:13.090445 6008 raft_consensus.cc:348] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.090575 6008 raft_consensus.cc:729] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.090751 6008 consensus_queue.cc:260] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.091068 6008 ts_tablet_manager.cc:1398] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:13.091547 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-7dbf7921ae84476ab53fbd4707f463eb received from {username='slave'} at 127.16.36.200:38023
I20240502 08:00:13.091627 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-7dbf7921ae84476ab53fbd4707f463eb on tablet 7dbf7921ae84476ab53fbd4707f463eb with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:13.126868 28790 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.129910 30477 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:00:13.130596 28397 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:00:13.131090 5817 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:00:13.132074 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d939e5035541a4b003c5d276493d5c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.132280 5817 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:13.134481 28529 consensus_queue.cc:237] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:13.136595 30346 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.136924 28790 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.137002 3943 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:00:13.139518 28463 consensus_peers.cc:485] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab785e07fa8248e5889bea8ac755d002. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.139477 5813 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.139640 3943 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:13.141973 28397 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.143576 30477 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.145481 28256 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.148933 3943 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:13.151473 28790 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:13.153146 30346 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:13.153659 28253 catalog_manager.cc:5484] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:13.197140 6165 ts_tablet_manager.cc:902] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:13.197777 6165 tablet_copy_client.cc:308] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:13.198362 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 3288b438832e488e8257227cb8934565 from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:00:13.198477 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 3288b438832e488e8257227cb8934565 from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-3288b438832e488e8257227cb8934565
I20240502 08:00:13.199633 28548 tablet_copy_source_session.cc:205] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.200670 6165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3288b438832e488e8257227cb8934565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.204100 6165 tablet_copy_client.cc:783] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.210000 6165 tablet_copy_client.cc:647] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.214422 6165 tablet_copy_client.cc:523] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.217937 6165 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:13.219599 28660 consensus_queue.cc:237] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:13.221516 30213 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.221904 28790 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.222550 5972 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:00:13.223587 28595 consensus_peers.cc:485] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.224215 5972 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:13.227772 5972 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:13.231248 28257 catalog_manager.cc:5484] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:13.231266 30213 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:13.231295 28790 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:13.240835 30346 consensus_queue.cc:237] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:13.242650 28398 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:13.242695 30213 raft_consensus.cc:1261] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:13.243093 2449 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:13.243427 2449 consensus_queue.cc:1035] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:00:13.244283 30279 consensus_peers.cc:485] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 -> Peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): Couldn't send request to peer c39bcb719db947b8afb0bb141e1f7a4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.260587 6226 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:13.263356 28398 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:13.267266 28255 catalog_manager.cc:5484] T 9c7e7dadff0a4cd08deb043a37a54424 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New cstate: current_term: 2 leader_uuid: "6efe17dcca7b4babad4785d19552b699" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:13.270153 6165 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.270488 6165 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:13.270684 6165 ts_tablet_manager.cc:1367] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.053s user 0.026s sys 0.003s
I20240502 08:00:13.270990 30213 raft_consensus.cc:2932] T 9c7e7dadff0a4cd08deb043a37a54424 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:13.271176 6165 raft_consensus.cc:348] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:13.271309 6165 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.271440 6165 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:13.285604 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.283689 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.292790 6008 ts_tablet_manager.cc:902] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:13.293282 6008 tablet_copy_client.cc:308] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:13.294814 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 0499a3f3e6bd4069bf992e32f1abfdb0 from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:55771)
I20240502 08:00:13.294929 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 0499a3f3e6bd4069bf992e32f1abfdb0 from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:55771: session id = e5e9553dabc94f218f6427c1685ffc38-0499a3f3e6bd4069bf992e32f1abfdb0
I20240502 08:00:13.295898 28812 tablet_copy_source_session.cc:205] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.297065 6008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0499a3f3e6bd4069bf992e32f1abfdb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.298404 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:13.299110 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:13.299557 6008 tablet_copy_client.cc:783] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.299713 6008 tablet_copy_client.cc:647] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.304131 6008 tablet_copy_client.cc:523] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.305533 6008 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:13.306640 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:13.324662 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.326859 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.337491 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:13.337957 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:13.339567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:13.353639 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.354725 28528 raft_consensus.cc:1203] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:00:13.279881 6165 ts_tablet_manager.cc:1398] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 08:00:13.362169 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-3288b438832e488e8257227cb8934565 received from {username='slave'} at 127.16.36.198:43585
I20240502 08:00:13.362263 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-3288b438832e488e8257227cb8934565 on tablet 3288b438832e488e8257227cb8934565 with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:13.368682 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.375725 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.379060 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.383947 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.416684 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.463956 6327 ts_tablet_manager.cc:902] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:13.464655 6327 tablet_copy_client.cc:308] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:13.465240 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet afc9c4d9dd5b4c039686ccd918616859 from peer de423605477a4819a9569821f59f03e4 ({username='slave'} at 127.16.36.196:59091)
I20240502 08:00:13.465342 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet afc9c4d9dd5b4c039686ccd918616859 from peer de423605477a4819a9569821f59f03e4 at {username='slave'} at 127.16.36.196:59091: session id = de423605477a4819a9569821f59f03e4-afc9c4d9dd5b4c039686ccd918616859
I20240502 08:00:13.466392 28681 tablet_copy_source_session.cc:205] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.469188 6008 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.470918 6008 tablet_bootstrap.cc:492] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:13.471129 6008 ts_tablet_manager.cc:1367] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.166s user 0.029s sys 0.000s
I20240502 08:00:13.471690 6008 raft_consensus.cc:348] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.471814 6008 raft_consensus.cc:729] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.471971 6008 consensus_queue.cc:260] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.472247 6008 ts_tablet_manager.cc:1398] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:00:13.475163 28812 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-0499a3f3e6bd4069bf992e32f1abfdb0 received from {username='slave'} at 127.16.36.200:55771
I20240502 08:00:13.475250 28812 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-0499a3f3e6bd4069bf992e32f1abfdb0 on tablet 0499a3f3e6bd4069bf992e32f1abfdb0 with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:13.478981 28528 consensus_queue.cc:237] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.480906 30477 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.481572 5378 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:00:13.482376 28463 consensus_peers.cc:485] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7fd7f8583e14c91a932767ebe23b419. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.482980 28790 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.483500 6327 ts_tablet_manager.cc:1875] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:13.484476 5378 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:13.491362 5378 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.495959 28248 catalog_manager.cc:5484] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.499132 30213 consensus_queue.cc:237] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:13.499963 28790 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.499970 30477 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.502804 30346 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:13.503187 28528 raft_consensus.cc:1261] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:13.504263 5998 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:13.505831 5998 consensus_queue.cc:1035] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:13.510191 5998 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.512974 28528 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.513804 28257 catalog_manager.cc:5484] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 6 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.516937 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet de90ee4ef90e4cb69ec37d7d9f0c86ef with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:13.520303 6327 ts_tablet_manager.cc:1888] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:00:13.520553 6327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afc9c4d9dd5b4c039686ccd918616859. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.523988 30346 raft_consensus.cc:2932] T de90ee4ef90e4cb69ec37d7d9f0c86ef P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.524443 6370 tablet_replica.cc:330] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:13.524633 6370 raft_consensus.cc:2229] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:13.524717 6327 tablet_copy_client.cc:783] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.524991 6327 tablet_copy_client.cc:647] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.526134 6327 tablet_copy_client.cc:523] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.529755 6327 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Bootstrap starting.
I20240502 08:00:13.536283 6370 raft_consensus.cc:2258] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:13.536994 6370 ts_tablet_manager.cc:1875] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:13.556255 6370 ts_tablet_manager.cc:1888] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:00:13.556504 6370 log.cc:1196] T de90ee4ef90e4cb69ec37d7d9f0c86ef P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/de90ee4ef90e4cb69ec37d7d9f0c86ef
I20240502 08:00:13.557179 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet de90ee4ef90e4cb69ec37d7d9f0c86ef (table rebalance_test_table_4 [id=e0a255a457004e77810fb9dc36cc3f28]) successfully deleted
I20240502 08:00:13.568840 6327 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.569187 6327 tablet_bootstrap.cc:492] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Bootstrap complete.
I20240502 08:00:13.569350 6327 ts_tablet_manager.cc:1367] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.001s
I20240502 08:00:13.569783 6327 raft_consensus.cc:348] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:13.569908 6327 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de423605477a4819a9569821f59f03e4, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.570061 6327 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:13.571259 6327 ts_tablet_manager.cc:1398] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:00:13.571785 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session de423605477a4819a9569821f59f03e4-afc9c4d9dd5b4c039686ccd918616859 received from {username='slave'} at 127.16.36.196:59091
I20240502 08:00:13.571870 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session de423605477a4819a9569821f59f03e4-afc9c4d9dd5b4c039686ccd918616859 on tablet afc9c4d9dd5b4c039686ccd918616859 with peer de423605477a4819a9569821f59f03e4
I20240502 08:00:13.574923 30608 raft_consensus.cc:1203] T 7dbf7921ae84476ab53fbd4707f463eb P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:00:13.576613 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.590112 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.593860 6421 ts_tablet_manager.cc:902] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:13.596706 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.598551 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:13.598899 6421 tablet_copy_client.cc:308] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:13.599399 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet ab785e07fa8248e5889bea8ac755d002 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:42745)
I20240502 08:00:13.599510 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet ab785e07fa8248e5889bea8ac755d002 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:42745: session id = 29126d5aee924f81bdd47cc7153420a1-ab785e07fa8248e5889bea8ac755d002
I20240502 08:00:13.600174 28790 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:13.600576 28548 tablet_copy_source_session.cc:205] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.601907 6421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab785e07fa8248e5889bea8ac755d002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.602566 28398 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:00:13.602922 5813 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:00:13.604501 28727 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d939e5035541a4b003c5d276493d5c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.604648 30477 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:00:13.604884 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21d939e5035541a4b003c5d276493d5c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.605068 5813 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:13.606091 6421 tablet_copy_client.cc:783] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.606256 6421 tablet_copy_client.cc:647] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.611932 5813 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.614279 28398 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.614377 30477 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.616187 6421 tablet_copy_client.cc:523] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.617309 28256 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.619040 6421 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:00:13.619949 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:13.640966 30346 raft_consensus.cc:1203] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.6->[2.7-2.7] Dedup: 2.7->[]
I20240502 08:00:13.643769 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.651206 6421 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.651551 6421 tablet_bootstrap.cc:492] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:00:13.651787 6421 ts_tablet_manager.cc:1367] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.033s user 0.024s sys 0.004s
I20240502 08:00:13.652246 6421 raft_consensus.cc:348] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:13.652379 6421 raft_consensus.cc:729] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.652544 6421 consensus_queue.cc:260] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:13.652841 6421 ts_tablet_manager.cc:1398] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:13.653328 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-ab785e07fa8248e5889bea8ac755d002 received from {username='slave'} at 127.16.36.199:42745
I20240502 08:00:13.653409 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-ab785e07fa8248e5889bea8ac755d002 on tablet ab785e07fa8248e5889bea8ac755d002 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:13.653417 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:13.653918 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.696022 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.700750 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.702582 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:13.711819 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:13.712113 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.717227 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:13.736099 6494 ts_tablet_manager.cc:902] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Initiating tablet copy from peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 08:00:13.736227 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.736685 6494 tablet_copy_client.cc:308] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.198:45421
I20240502 08:00:13.737159 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.737299 30366 tablet_copy_service.cc:140] P 6efe17dcca7b4babad4785d19552b699: Received BeginTabletCopySession request for tablet 9c7e7dadff0a4cd08deb043a37a54424 from peer c39bcb719db947b8afb0bb141e1f7a4e ({username='slave'} at 127.16.36.194:52473)
I20240502 08:00:13.737366 30366 tablet_copy_service.cc:161] P 6efe17dcca7b4babad4785d19552b699: Beginning new tablet copy session on tablet 9c7e7dadff0a4cd08deb043a37a54424 from peer c39bcb719db947b8afb0bb141e1f7a4e at {username='slave'} at 127.16.36.194:52473: session id = c39bcb719db947b8afb0bb141e1f7a4e-9c7e7dadff0a4cd08deb043a37a54424
I20240502 08:00:13.738260 30366 tablet_copy_source_session.cc:205] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.740955 6494 ts_tablet_manager.cc:1875] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:13.745576 6494 ts_tablet_manager.cc:1888] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 08:00:13.746237 6494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c7e7dadff0a4cd08deb043a37a54424. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.773311 6493 ts_tablet_manager.cc:902] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:13.773955 6493 tablet_copy_client.cc:308] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:13.774493 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:45489)
I20240502 08:00:13.774608 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 2ba62cb7ab7b429d85e0ac037c127cb4 from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:45489: session id = 71922d97e1a4414698ecd14a7b2dedec-2ba62cb7ab7b429d85e0ac037c127cb4
I20240502 08:00:13.775558 28681 tablet_copy_source_session.cc:205] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.775919 6494 tablet_copy_client.cc:783] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.776134 6494 tablet_copy_client.cc:647] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.776892 6493 ts_tablet_manager.cc:1875] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:13.777632 6494 tablet_copy_client.cc:523] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.779088 6494 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 08:00:13.779729 6493 ts_tablet_manager.cc:1888] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:00:13.779970 6493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ba62cb7ab7b429d85e0ac037c127cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:13.781668 6493 tablet_copy_client.cc:783] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:13.781903 6493 tablet_copy_client.cc:647] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:13.785073 6493 tablet_copy_client.cc:523] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:13.786700 6493 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:00:13.811331 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.818976 30608 raft_consensus.cc:1203] T 0499a3f3e6bd4069bf992e32f1abfdb0 P e5e9553dabc94f218f6427c1685ffc38 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 08:00:13.832262 6493 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.832680 6493 tablet_bootstrap.cc:492] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:00:13.832904 6493 ts_tablet_manager.cc:1367] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.046s user 0.028s sys 0.001s
I20240502 08:00:13.833494 6493 raft_consensus.cc:348] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:13.833652 6493 raft_consensus.cc:729] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.833896 6493 consensus_queue.cc:260] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:13.834355 6493 ts_tablet_manager.cc:1398] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:00:13.834939 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-2ba62cb7ab7b429d85e0ac037c127cb4 received from {username='slave'} at 127.16.36.193:45489
I20240502 08:00:13.835032 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-2ba62cb7ab7b429d85e0ac037c127cb4 on tablet 2ba62cb7ab7b429d85e0ac037c127cb4 with peer 71922d97e1a4414698ecd14a7b2dedec
I20240502 08:00:13.837610 28398 consensus_queue.cc:237] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 4, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:13.839179 28528 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:13.839351 28790 raft_consensus.cc:1261] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:13.839684 5263 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:13.839943 5263 consensus_queue.cc:1035] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240502 08:00:13.841297 28334 consensus_peers.cc:485] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57f7c358ace742649df52380939ece9d. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.847628 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:13.850116 6544 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.852048 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:13.853111 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.854478 28528 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.856112 28255 catalog_manager.cc:5484] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.858359 28790 raft_consensus.cc:2932] T 57f7c358ace742649df52380939ece9d P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:00:13.861313 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:13.865336 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:13.870945 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:13.875895 6494 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:13.876411 6494 tablet_bootstrap.cc:492] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap complete.
I20240502 08:00:13.876667 6494 ts_tablet_manager.cc:1367] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.098s user 0.031s sys 0.001s
I20240502 08:00:13.877111 6494 raft_consensus.cc:348] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:13.877226 6494 raft_consensus.cc:729] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 08:00:13.877362 6494 consensus_queue.cc:260] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:13.877667 6494 ts_tablet_manager.cc:1398] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:00:13.878299 30366 tablet_copy_service.cc:342] P 6efe17dcca7b4babad4785d19552b699: Request end of tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-9c7e7dadff0a4cd08deb043a37a54424 received from {username='slave'} at 127.16.36.194:52473
I20240502 08:00:13.878388 30366 tablet_copy_service.cc:433] P 6efe17dcca7b4babad4785d19552b699: ending tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-9c7e7dadff0a4cd08deb043a37a54424 on tablet 9c7e7dadff0a4cd08deb043a37a54424 with peer c39bcb719db947b8afb0bb141e1f7a4e
I20240502 08:00:13.902763 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:13.927959 28790 raft_consensus.cc:1203] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:00:13.938063 28660 consensus_queue.cc:237] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:13.940176 28398 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.940833 6591 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:13.941604 30477 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:13.943151 6591 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 08:00:13.943565 28594 consensus_peers.cc:485] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d08ad2a981fc4c13bc89d22feea7d556. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.945302 6591 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.948323 28398 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.948854 28256 catalog_manager.cc:5484] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.953176 30477 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:13.964272 30213 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:13.965620 30346 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 08:00:13.966436 6367 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:00:13.969619 5998 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.970223 30346 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 9 to 10, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.973066 28257 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 9 to 10, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 3 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:13.976735 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 5d1c1fa046b741aa8cf296aa57847fe3 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:13.977452 6638 tablet_replica.cc:330] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:13.977612 6638 raft_consensus.cc:2229] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:13.977766 6638 raft_consensus.cc:2258] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:13.978502 6638 ts_tablet_manager.cc:1875] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:13.984675 6638 ts_tablet_manager.cc:1888] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.9
I20240502 08:00:13.984792 6638 log.cc:1196] T 5d1c1fa046b741aa8cf296aa57847fe3 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/5d1c1fa046b741aa8cf296aa57847fe3
I20240502 08:00:13.985916 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 5d1c1fa046b741aa8cf296aa57847fe3 (table rebalance_test_table_6 [id=77d8070fb35f4aeaa0aba32c29ec61cf]) successfully deleted
I20240502 08:00:13.986052 6643 ts_tablet_manager.cc:902] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:13.986557 6643 tablet_copy_client.cc:308] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:13.987278 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 21d939e5035541a4b003c5d276493d5c from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:47747)
I20240502 08:00:13.987380 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 21d939e5035541a4b003c5d276493d5c from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:47747: session id = 6efe17dcca7b4babad4785d19552b699-21d939e5035541a4b003c5d276493d5c
I20240502 08:00:13.988394 28812 tablet_copy_source_session.cc:205] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:13.992939 30213 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
W20240502 08:00:13.994587 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5d1c1fa046b741aa8cf296aa57847fe3. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:13.994693 30346 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:00:13.995069 5998 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:00:13.996703 6367 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:14.000054 28244 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 5d1c1fa046b741aa8cf296aa57847fe3 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:14.000416 6668 ts_tablet_manager.cc:902] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:14.000527 30346 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:14.001184 6668 tablet_copy_client.cc:308] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:14.001945 28812 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 21d939e5035541a4b003c5d276493d5c from peer e5e9553dabc94f218f6427c1685ffc38 ({username='slave'} at 127.16.36.200:55771)
I20240502 08:00:14.002059 28812 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 21d939e5035541a4b003c5d276493d5c from peer e5e9553dabc94f218f6427c1685ffc38 at {username='slave'} at 127.16.36.200:55771: session id = e5e9553dabc94f218f6427c1685ffc38-21d939e5035541a4b003c5d276493d5c
I20240502 08:00:14.002178 28256 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) added. New cstate: current_term: 3 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } }
I20240502 08:00:14.003036 28812 tablet_copy_source_session.cc:205] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:14.004094 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:14.004739 6643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21d939e5035541a4b003c5d276493d5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:14.006947 6668 tablet_copy_client.cc:783] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:14.006951 6643 tablet_copy_client.cc:783] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:14.007112 6643 tablet_copy_client.cc:647] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:14.007248 6668 tablet_copy_client.cc:647] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:14.010893 6643 tablet_copy_client.cc:523] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:14.010267 6668 tablet_copy_client.cc:523] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:14.012342 6643 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:14.013583 6668 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: Bootstrap starting.
I20240502 08:00:14.048049 30477 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 8, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:14.050570 28790 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 08:00:14.051014 30213 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240502 08:00:14.052389 5693 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:00:14.052703 30412 consensus_peers.cc:485] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 -> Peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): Couldn't send request to peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.052825 5693 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:00:14.061586 30215 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:14.062601 6709 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 3 LEADER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:14.065274 28255 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New cstate: current_term: 3 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:14.073124 28790 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Committing config change with OpId 3.10: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:14.078676 30477 raft_consensus.cc:1203] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 08:00:14.092573 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.097487 6643 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:14.097887 6643 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:14.098116 6643 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.086s user 0.029s sys 0.004s
I20240502 08:00:14.098625 6643 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:14.098767 6643 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:14.098945 6643 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:14.095314 6668 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:14.099378 6643 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 08:00:14.099555 6668 tablet_bootstrap.cc:492] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: Bootstrap complete.
I20240502 08:00:14.099887 28811 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-21d939e5035541a4b003c5d276493d5c received from {username='slave'} at 127.16.36.198:47747
I20240502 08:00:14.099982 28811 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-21d939e5035541a4b003c5d276493d5c on tablet 21d939e5035541a4b003c5d276493d5c with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:14.100033 6668 ts_tablet_manager.cc:1367] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: Time spent bootstrapping tablet: real 0.086s user 0.029s sys 0.003s
I20240502 08:00:14.100529 6668 raft_consensus.cc:348] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:14.100667 6668 raft_consensus.cc:729] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e9553dabc94f218f6427c1685ffc38, State: Initialized, Role: FOLLOWER
I20240502 08:00:14.101310 6668 consensus_queue.cc:260] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:14.102622 6668 ts_tablet_manager.cc:1398] T 21d939e5035541a4b003c5d276493d5c P e5e9553dabc94f218f6427c1685ffc38: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:00:14.103111 28811 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session e5e9553dabc94f218f6427c1685ffc38-21d939e5035541a4b003c5d276493d5c received from {username='slave'} at 127.16.36.200:55771
I20240502 08:00:14.103192 28811 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session e5e9553dabc94f218f6427c1685ffc38-21d939e5035541a4b003c5d276493d5c on tablet 21d939e5035541a4b003c5d276493d5c with peer e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:14.106567 6668 ts_tablet_manager.cc:902] T a7fd7f8583e14c91a932767ebe23b419 P e5e9553dabc94f218f6427c1685ffc38: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
F20240502 08:00:14.106763 6668 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.queue_time_us, tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.queue_time_us) (tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_cpu_time_us, tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_cpu_time_us) (tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_wall_time_us, tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_wall_time_us) (tablet-download-pool-07461c988fb3402c91cfd9235efbb9b1.queue_time_us, tablet-download-pool-07461c988fb3402c91cfd9235efbb9b1.queue_time_us) (tablet-download-pool-07461c988fb3402c91cfd9235efbb9b1.run_cpu_time_us, tablet-download-pool-07461c988fb3402c91cfd9235efbb9b1.run_cpu_time_us) (tablet-download-pool-07461c988fb3402c91cfd9235efbb9b1.run_wall_time_us, tablet-download-pool-07461c988fb3402c91cfd9235efbb9b1.run_wall_time_us) (tablet-download-pool-21d939e5035541a4b003c5d276493d5c.queue_time_us, tablet-download-pool-21d939e5035541a4b003c5d276493d5c.queue_time_us) (tablet-download-pool-21d939e5035541a4b003c5d276493d5c.run_cpu_time_us, tablet-download-pool-21d939e5035541a4b003c5d276493d5c.run_cpu_time_us) (tablet-download-pool-21d939e5035541a4b003c5d276493d5c.run_wall_time_us, tablet-download-pool-21d939e5035541a4b003c5d276493d5c.run_wall_time_us) (tablet-download-pool-3288b438832e488e8257227cb8934565.queue_time_us, tablet-download-pool-3288b438832e488e8257227cb8934565.queue_time_us) (tablet-download-pool-3288b438832e488e8257227cb8934565.run_cpu_time_us, tablet-download-pool-3288b438832e488e8257227cb8934565.run_cpu_time_us) (tablet-download-pool-3288b438832e488e8257227cb8934565.run_wall_time_us, tablet-download-pool-3288b438832e488e8257227cb8934565.run_wall_time_us) (tablet-download-pool-3598dbda342f49aea2384b5beb17d7a1.queue_time_us, tablet-download-pool-3598dbda342f49aea2384b5beb17d7a1.queue_time_us) (tablet-download-pool-3598dbda342f49aea2384b5beb17d7a1.run_cpu_time_us, tablet-download-pool-3598dbda342f49aea2384b5beb17d7a1.run_cpu_time_us) (tablet-download-pool-3598dbda342f49aea2384b5beb17d7a1.run_wall_time_us, tablet-download-pool-3598dbda342f49aea2384b5beb17d7a1.run_wall_time_us) (tablet-download-pool-4cb29b8b566747e5b82a8969a070123e.queue_time_us, tablet-download-pool-4cb29b8b566747e5b82a8969a070123e.queue_time_us) (tablet-download-pool-4cb29b8b566747e5b82a8969a070123e.run_cpu_time_us, tablet-download-pool-4cb29b8b566747e5b82a8969a070123e.run_cpu_time_us) (tablet-download-pool-4cb29b8b566747e5b82a8969a070123e.run_wall_time_us, tablet-download-pool-4cb29b8b566747e5b82a8969a070123e.run_wall_time_us) (tablet-download-pool-57f7c358ace742649df52380939ece9d.queue_time_us, tablet-download-pool-57f7c358ace742649df52380939ece9d.queue_time_us) (tablet-download-pool-57f7c358ace742649df52380939ece9d.run_cpu_time_us, tablet-download-pool-57f7c358ace742649df52380939ece9d.run_cpu_time_us) (tablet-download-pool-57f7c358ace742649df52380939ece9d.run_wall_time_us, tablet-download-pool-57f7c358ace742649df52380939ece9d.run_wall_time_us) (tablet-download-pool-698f494a22d34e659766b97cb24d0293.queue_time_us, tablet-download-pool-698f494a22d34e659766b97cb24d0293.queue_time_us) (tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_cpu_time_us, tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_cpu_time_us) (tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_wall_time_us, tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_wall_time_us) (tablet-download-pool-6a40382745f84f0eadf9e0ee2e057289.queue_time_us, tablet-download-pool-6a40382745f84f0eadf9e0ee2e057289.queue_time_us) (tablet-download-pool-6a40382745f84f0eadf9e0ee2e057289.run_cpu_time_us, tablet-download-pool-6a40382745f84f0eadf9e0ee2e057289.run_cpu_time_us) (tablet-download-pool-6a40382745f84f0eadf9e0ee2e057289.run_wall_time_us, tablet-download-pool-6a40382745f84f0eadf9e0ee2e057289.run_wall_time_us) (tablet-download-pool-7dbf7921ae84476ab53fbd4707f463eb.queue_time_us, tablet-download-pool-7dbf7921ae84476ab53fbd4707f463eb.queue_time_us) (tablet-download-pool-7dbf7921ae84476ab53fbd4707f463eb.run_cpu_time_us, tablet-download-pool-7dbf7921ae84476ab53fbd4707f463eb.run_cpu_time_us) (tablet-download-pool-7dbf7921ae84476ab53fbd4707f463eb.run_wall_time_us, tablet-download-pool-7dbf7921ae84476ab53fbd4707f463eb.run_wall_time_us) (tablet-download-pool-9e1cd0ec72d1429f9621d217b1bf845a.queue_time_us, tablet-download-pool-9e1cd0ec72d1429f9621d217b1bf845a.queue_time_us) (tablet-download-pool-9e1cd0ec72d1429f9621d217b1bf845a.run_cpu_time_us, tablet-download-pool-9e1cd0ec72d1429f9621d217b1bf845a.run_cpu_time_us) (tablet-download-pool-9e1cd0ec72d1429f9621d217b1bf845a.run_wall_time_us, tablet-download-pool-9e1cd0ec72d1429f9621d217b1bf845a.run_wall_time_us) (tablet-download-pool-a7fd7f8583e14c91a932767ebe23b419.queue_time_us, tablet-download-pool-a7fd7f8583e14c91a932767ebe23b419.queue_time_us) (tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.queue_time_us, tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.queue_time_us) (tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_cpu_time_us, tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_cpu_time_us) (tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_wall_time_us, tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_wall_time_us) (tablet-download-pool-f98044a1730e4a04a7088199d4f0850e.queue_time_us, tablet-download-pool-f98044a1730e4a04a7088199d4f0850e.queue_time_us) (tablet-download-pool-f98044a1730e4a04a7088199d4f0850e.run_cpu_time_us, tablet-download-pool-f98044a1730e4a04a7088199d4f0850e.run_cpu_time_us) (tablet-download-pool-f98044a1730e4a04a7088199d4f0850e.run_wall_time_us, tablet-download-pool-f98044a1730e4a04a7088199d4f0850e.run_wall_time_us) (tablet-download-pool-fddc382ab3164ccdaa86f802c83da8be.queue_time_us, tablet-download-pool-fddc382ab3164ccdaa86f802c83da8be.queue_time_us) (tablet-download-pool-fddc382ab3164ccdaa86f802c83da8be.run_cpu_time_us, tablet-download-pool-fddc382ab3164ccdaa86f802c83da8be.run_cpu_time_us) (tablet-download-pool-fddc382ab3164ccdaa86f802c83da8be.run_wall_time_us, tablet-download-pool-fddc382ab3164ccdaa86f802c83da8be.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f85c687ddcd google::LogMessage::Fail() at ??:0
I20240502 08:00:14.127219 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f85c6881b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f85c687d7cc google::LogMessage::Flush() at ??:0
@ 0x7f85c687ef59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 08:00:14.132380 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
@ 0x7f85c7ce7044 kudu::TraceMetrics::InternName() at ??:0
I20240502 08:00:14.135339 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f85c7ccbf98 kudu::ThreadPool::ThreadPool() at ??:0
I20240502 08:00:14.146616 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:14.151648 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f85c7cc9efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f85ca87c77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 08:00:14.167131 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
@ 0x7f85ca885a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 08:00:14.173015 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
@ 0x7f85ca90676f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240502 08:00:14.183104 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:14.185027 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:14.199084 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.203121 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:14.206377 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.217123 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.218515 28398 raft_consensus.cc:1203] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
@ 0x7f85ca91a4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 08:00:14.231362 28528 raft_consensus.cc:1203] T 9c7e7dadff0a4cd08deb043a37a54424 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 08:00:14.233377 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:14.234825 30588 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
@ 0x7f85ca905226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f85ca914c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55579e71f75c std::function<>::operator()() at ??:0
@ 0x7f85c7ccee06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f85c7ccf6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f85c7cd103b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55579e71f75c std::function<>::operator()() at ??:0
@ 0x7f85c7cbe807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f85c84fc6db start_thread at ??:0
@ 0x7f85c5a3471f clone at ??:0
I20240502 08:00:14.302821 30215 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 3.11, Last appended by leader: 7, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:14.304934 30346 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20240502 08:00:14.305436 6367 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 08:00:14.306740 30149 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.355307 28790 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:14.357934 30477 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:00:14.358619 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): Couldn't send request to peer c39bcb719db947b8afb0bb141e1f7a4e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.359450 5817 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:14.382792 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:14.387766 6643 ts_tablet_manager.cc:902] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:00:14.388514 6643 tablet_copy_client.cc:308] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:00:14.388947 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 57f7c358ace742649df52380939ece9d from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:56805)
I20240502 08:00:14.389058 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 57f7c358ace742649df52380939ece9d from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:56805: session id = 6efe17dcca7b4babad4785d19552b699-57f7c358ace742649df52380939ece9d
I20240502 08:00:14.390056 28419 tablet_copy_source_session.cc:205] T 57f7c358ace742649df52380939ece9d P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:14.391732 6643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f7c358ace742649df52380939ece9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:14.392954 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.395715 6643 tablet_copy_client.cc:783] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:14.395893 6643 tablet_copy_client.cc:647] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:14.399156 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.410233 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:14.410305 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:14.425781 6643 tablet_copy_client.cc:523] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:14.427392 6643 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:00:14.430662 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:14.460651 6643 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:14.460971 6643 tablet_bootstrap.cc:492] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:00:14.461287 6643 ts_tablet_manager.cc:1367] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.001s
I20240502 08:00:14.461823 6643 raft_consensus.cc:348] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:14.461956 6643 raft_consensus.cc:729] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:00:14.462122 6643 consensus_queue.cc:260] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:00:14.462436 6643 ts_tablet_manager.cc:1398] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:00:14.462956 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-57f7c358ace742649df52380939ece9d received from {username='slave'} at 127.16.36.198:56805
I20240502 08:00:14.463038 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-57f7c358ace742649df52380939ece9d on tablet 57f7c358ace742649df52380939ece9d with peer 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:14.472312 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:14.475478 30346 raft_consensus.cc:1203] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:00:14.509974 6884 ts_tablet_manager.cc:902] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Initiating tablet copy from peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589)
I20240502 08:00:14.510586 6884 tablet_copy_client.cc:308] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.199:43589
I20240502 08:00:14.511276 30497 tablet_copy_service.cc:140] P 29126d5aee924f81bdd47cc7153420a1: Received BeginTabletCopySession request for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 ({username='slave'} at 127.16.36.195:52903)
I20240502 08:00:14.511404 30497 tablet_copy_service.cc:161] P 29126d5aee924f81bdd47cc7153420a1: Beginning new tablet copy session on tablet 04d358245cdf4b2ebb0b5005f75cf1e0 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 at {username='slave'} at 127.16.36.195:52903: session id = ed3ca7e829bc42b5a8e9a7d1bdcc5a44-04d358245cdf4b2ebb0b5005f75cf1e0
I20240502 08:00:14.512895 30497 tablet_copy_source_session.cc:205] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:14.514205 6884 ts_tablet_manager.cc:1875] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:14.518594 6884 ts_tablet_manager.cc:1888] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:00:14.518823 6884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d358245cdf4b2ebb0b5005f75cf1e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:14.521018 6884 tablet_copy_client.cc:783] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:14.521167 6884 tablet_copy_client.cc:647] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:14.523864 6884 tablet_copy_client.cc:523] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:14.525552 6884 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 08:00:14.527350 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.551725 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.552021 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:14.560941 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:14.563714 6884 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:14.564042 6884 tablet_bootstrap.cc:492] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap complete.
I20240502 08:00:14.564219 6884 ts_tablet_manager.cc:1367] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.039s user 0.030s sys 0.000s
I20240502 08:00:14.564680 6884 raft_consensus.cc:348] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:14.564808 6884 raft_consensus.cc:729] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 08:00:14.564975 6884 consensus_queue.cc:260] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 3.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:14.565872 6884 ts_tablet_manager.cc:1398] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:00:14.566443 30497 tablet_copy_service.cc:342] P 29126d5aee924f81bdd47cc7153420a1: Request end of tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-04d358245cdf4b2ebb0b5005f75cf1e0 received from {username='slave'} at 127.16.36.195:52903
I20240502 08:00:14.566536 30497 tablet_copy_service.cc:433] P 29126d5aee924f81bdd47cc7153420a1: ending tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-04d358245cdf4b2ebb0b5005f75cf1e0 on tablet 04d358245cdf4b2ebb0b5005f75cf1e0 with peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:00:14.589026 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:14.593216 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:14.603667 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:14.664710 28725 connection.cc:535] client connection to 127.16.36.200:43189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.664768 28594 connection.cc:535] client connection to 127.16.36.200:43189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.664841 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.664896 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.665251 28332 connection.cc:535] client connection to 127.16.36.200:43189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.665364 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.665437 28463 connection.cc:535] client connection to 127.16.36.200:43189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.665529 30148 connection.cc:535] client connection to 127.16.36.200:43189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.665568 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.665625 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:00:14.667238 6909 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:00:14.666371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.200:43189 (local address 127.16.36.196:55419)
0502 08:00:14.666776 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0502 08:00:14.666836 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":54,"threads_started":1}
W20240502 08:00:14.667352 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.667982 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.668614 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:14.668697 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.668751 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.668795 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.669652 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.669838 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.669914 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:14.669963 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:14.670364 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.693902 30659 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:58.416245 30662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:58.428210 30662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:58.433403 30662 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:59.550459 30662 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:00:00.157631 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.157758 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.157892 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.157943 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.157989 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.158059 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.158104 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.158151 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.158195 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.158241 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.158285 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.158361 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.158406 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.158449 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.158504 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:00.202982 30662 rebalancer_tool.cc:1074] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:00.286839 30662 rebalancer_tool.cc:1074] tablet 1631d3e38209497ea2454fdfc555a319: '71922d97e1a4414698ecd14a7b2dedec' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:00.450909 30662 rebalancer_tool.cc:1074] tablet 4cb29b8b566747e5b82a8969a070123e: 'c39bcb719db947b8afb0bb141e1f7a4e' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:00.516746 30662 rebalancer_tool.cc:1074] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.528595 30662 rebalancer_tool.cc:1084] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move ignored: Illegal state: tablet 9e1cd0ec72d1429f9621d217b1bf845a: replica 29126d5aee924f81bdd47cc7153420a1 is already present
I20240502 08:00:00.528695 30662 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:00.874390 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.874491 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.874522 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.874557 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.874581 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.874631 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.874672 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.874697 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.874732 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.874770 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.874816 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.874855 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.874887 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.874917 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.874944 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.874969 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.875000 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.875025 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.875052 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.875082 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.875113 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:01.211367 30662 rebalancer_tool.cc:1074] tablet 14bbfb3758e04aeab0df4bbab8380061: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:01.352530 30662 rebalancer_tool.cc:1074] tablet ab785e07fa8248e5889bea8ac755d002: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:01.473954 30662 rebalancer_tool.cc:1074] tablet f98044a1730e4a04a7088199d4f0850e: 'de423605477a4819a9569821f59f03e4' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:01.609617 30662 rebalancer_tool.cc:1074] tablet 21d939e5035541a4b003c5d276493d5c: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:01.852658 30662 rebalancer_tool.cc:1074] tablet dcd68a21ed7b4205a2f0cf75f80df0e6: '71922d97e1a4414698ecd14a7b2dedec' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:01.992924 30662 rebalancer_tool.cc:1084] tablet df20642aef7748ff8962b3095e1ab56c: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:00:01.993013 30662 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:02.537869 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server e5e9553dabc94f218f6427c1685ffc38 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.544839 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.544929 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.545058 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.545251 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.545336 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.545409 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.545470 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.545554 30662 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:02.568281 30662 rebalancer_tool.cc:1074] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:02.589473 30662 rebalancer_tool.cc:1074] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:02.609848 30662 rebalancer_tool.cc:1074] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:02.620798 30662 rebalancer_tool.cc:1074] tablet 698f494a22d34e659766b97cb24d0293: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:02.641945 30662 rebalancer_tool.cc:1074] tablet 57f7c358ace742649df52380939ece9d: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
W20240502 08:00:05.397854 1659 outbound_call.cc:322] RPC callback for RPC call kudu.server.GenericService.GetStatus -> {remote=127.16.36.193:33537, user_credentials={real_user=slave}} blocked reactor thread for 53154.2us
I20240502 08:00:08.742374 30662 rebalancer_tool.cc:1123] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move is abandoned: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:08.783862 30662 rebalancer_tool.cc:1136] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:11.830216 30662 rebalancer_tool.cc:1136] tablet 14bbfb3758e04aeab0df4bbab8380061: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:12.524922 30662 rebalancer_tool.cc:1136] tablet 698f494a22d34e659766b97cb24d0293: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:12.592497 30662 rebalancer_tool.cc:1136] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:12.626545 30662 rebalancer_tool.cc:1136] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:12.640921 30662 rebalancer_tool.cc:1136] tablet dcd68a21ed7b4205a2f0cf75f80df0e6: '71922d97e1a4414698ecd14a7b2dedec' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:12.655753 30662 rebalancer_tool.cc:1136] tablet 21d939e5035541a4b003c5d276493d5c: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:12.675163 30662 rebalancer_tool.cc:1136] tablet f98044a1730e4a04a7088199d4f0850e: 'de423605477a4819a9569821f59f03e4' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:12.700691 30662 rebalancer_tool.cc:1136] tablet ab785e07fa8248e5889bea8ac755d002: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:12.738046 30662 rebalancer_tool.cc:1136] tablet 4cb29b8b566747e5b82a8969a070123e: 'c39bcb719db947b8afb0bb141e1f7a4e' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:12.745995 30662 rebalancer_tool.cc:1136] tablet 1631d3e38209497ea2454fdfc555a319: '71922d97e1a4414698ecd14a7b2dedec' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:13.095263 30662 rebalancer_tool.cc:1074] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:13.104754 30662 rebalancer_tool.cc:1074] tablet c65fe7f823df4b4996769fc784d3adb7: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:13.133219 30662 rebalancer_tool.cc:1074] tablet 3598dbda342f49aea2384b5beb17d7a1: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:13.180254 30662 rebalancer_tool.cc:1136] tablet 3598dbda342f49aea2384b5beb17d7a1: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:13.195113 30662 rebalancer_tool.cc:1136] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:13.215942 30662 rebalancer_tool.cc:1136] tablet 57f7c358ace742649df52380939ece9d: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:13.258807 30662 rebalancer_tool.cc:1136] tablet c65fe7f823df4b4996769fc784d3adb7: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:13.986819 30662 rebalancer_tool.cc:1074] tablet a28d406b43194bc6b775707d021618f0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:14.051103 30662 rebalancer_tool.cc:1074] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:14.061568 30662 rebalancer_tool.cc:1074] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:14.111714 30662 rebalancer_tool.cc:1136] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:14.263700 30662 rebalancer_tool.cc:1136] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:14.288902 30662 rebalancer_tool.cc:1136] tablet a28d406b43194bc6b775707d021618f0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:14.314152 30662 rebalancer_tool.cc:1136] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
W20240502 08:00:14.668282 6807 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:00:14.578929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.200:43189 (local address 127.0.0.1:33758)
0502 08:00:14.583812 (+ 4883us) negotiation.cc:107] Waiting for socket to connect
0502 08:00:14.583827 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 08:00:14.583889 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:00:14.667627 (+ 83738us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":4811,"thread_start_us":68,"threads_started":1}
Illegal state: tablet server e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): unacceptable health status UNAVAILABLE
W20240502 08:00:14.700203 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:14.711966 28660 consensus_queue.cc:237] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:14.713405 30213 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:14.713850 28790 raft_consensus.cc:1261] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:14.713865 6600 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:14.714488 6600 consensus_queue.cc:1035] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:14.717010 6600 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:14.718725 28790 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:14.718758 30213 raft_consensus.cc:2932] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:14.721326 28249 catalog_manager.cc:5484] T 2ba62cb7ab7b429d85e0ac037c127cb4 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 6 to 7, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:14.727663 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ba62cb7ab7b429d85e0ac037c127cb4 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:14.728828 6972 tablet_replica.cc:330] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:14.729223 6972 raft_consensus.cc:2229] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:14.729368 6972 raft_consensus.cc:2258] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:14.730422 6972 ts_tablet_manager.cc:1875] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:00:14.732352 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.733381 6972 ts_tablet_manager.cc:1888] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:00:14.733582 6972 log.cc:1196] T 2ba62cb7ab7b429d85e0ac037c127cb4 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ba62cb7ab7b429d85e0ac037c127cb4
I20240502 08:00:14.734166 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 2ba62cb7ab7b429d85e0ac037c127cb4 (table rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]) successfully deleted
I20240502 08:00:14.759414 30457 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.762073 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:14.767223 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:14.781051 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:14.789710 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.790935 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:14.802814 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:14.813978 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.824431 6996 ts_tablet_manager.cc:902] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:14.828099 6996 tablet_copy_client.cc:308] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:14.828694 28811 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet 6a40382745f84f0eadf9e0ee2e057289 from peer c39bcb719db947b8afb0bb141e1f7a4e ({username='slave'} at 127.16.36.194:51335)
I20240502 08:00:14.828825 28811 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet 6a40382745f84f0eadf9e0ee2e057289 from peer c39bcb719db947b8afb0bb141e1f7a4e at {username='slave'} at 127.16.36.194:51335: session id = c39bcb719db947b8afb0bb141e1f7a4e-6a40382745f84f0eadf9e0ee2e057289
I20240502 08:00:14.830025 28811 tablet_copy_source_session.cc:205] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 08:00:14.830524 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:14.831228 6996 ts_tablet_manager.cc:1875] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:14.833634 6996 ts_tablet_manager.cc:1888] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 08:00:14.833811 6996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a40382745f84f0eadf9e0ee2e057289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:14.835290 6996 tablet_copy_client.cc:783] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:14.835476 6996 tablet_copy_client.cc:647] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:14.836665 6996 tablet_copy_client.cc:523] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:14.837818 6996 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 08:00:14.839802 28398 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a28d406b43194bc6b775707d021618f0"
dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42982
I20240502 08:00:14.839960 28398 raft_consensus.cc:595] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Received request to transfer leadership
W20240502 08:00:14.846549 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:14.869386 6996 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:00:14.869876 6996 tablet_bootstrap.cc:492] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap complete.
I20240502 08:00:14.870108 6996 ts_tablet_manager.cc:1367] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.002s
I20240502 08:00:14.870632 6996 raft_consensus.cc:348] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:14.870954 6996 raft_consensus.cc:729] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 08:00:14.871099 6996 consensus_queue.cc:260] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:00:14.871415 6996 ts_tablet_manager.cc:1398] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:00:14.872009 28811 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-6a40382745f84f0eadf9e0ee2e057289 received from {username='slave'} at 127.16.36.194:51335
I20240502 08:00:14.872097 28811 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-6a40382745f84f0eadf9e0ee2e057289 on tablet 6a40382745f84f0eadf9e0ee2e057289 with peer c39bcb719db947b8afb0bb141e1f7a4e
I20240502 08:00:14.884474 7043 ts_tablet_manager.cc:902] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:14.885511 7043 tablet_copy_client.cc:308] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.197:35407
I20240502 08:00:14.887071 28660 raft_consensus.cc:1203] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.9->[3.10-3.10] Dedup: 3.10->[]
I20240502 08:00:14.887573 30235 tablet_copy_service.cc:140] P 6c68c916f39a45f697df7eaf8973225a: Received BeginTabletCopySession request for tablet 5d1c1fa046b741aa8cf296aa57847fe3 from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:58623)
I20240502 08:00:14.887706 30235 tablet_copy_service.cc:161] P 6c68c916f39a45f697df7eaf8973225a: Beginning new tablet copy session on tablet 5d1c1fa046b741aa8cf296aa57847fe3 from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:58623: session id = 71922d97e1a4414698ecd14a7b2dedec-5d1c1fa046b741aa8cf296aa57847fe3
I20240502 08:00:14.891942 30235 tablet_copy_source_session.cc:205] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:14.893517 7043 ts_tablet_manager.cc:1875] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:14.893697 6548 raft_consensus.cc:979] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: : Instructing follower 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:14.893788 6548 raft_consensus.cc:1067] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Signalling peer 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:14.894587 30477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a28d406b43194bc6b775707d021618f0"
dest_uuid: "29126d5aee924f81bdd47cc7153420a1"
from {username='slave'} at 127.16.36.193:46783
I20240502 08:00:14.894722 30477 raft_consensus.cc:482] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:14.894780 30477 raft_consensus.cc:3037] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:14.895589 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:14.895785 30477 raft_consensus.cc:504] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:14.896136 30477 leader_election.cc:288] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 2 election: Requested vote from peers 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:14.896792 7043 ts_tablet_manager.cc:1888] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:00:14.897027 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d1c1fa046b741aa8cf296aa57847fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:14.898021 28398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28d406b43194bc6b775707d021618f0" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
I20240502 08:00:14.898137 28398 raft_consensus.cc:3032] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:14.898175 28398 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 08:00:14.898313 28398 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:14.898480 28398 raft_consensus.cc:3037] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:14.899174 7043 tablet_copy_client.cc:783] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:14.899302 7043 tablet_copy_client.cc:647] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:14.899549 28398 raft_consensus.cc:2454] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:14.899991 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a28d406b43194bc6b775707d021618f0" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:14.900118 30213 raft_consensus.cc:3037] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:14.901161 30213 raft_consensus.cc:2454] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:14.902571 30413 leader_election.cc:302] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29126d5aee924f81bdd47cc7153420a1, 6c68c916f39a45f697df7eaf8973225a; no voters:
I20240502 08:00:14.902784 6734 raft_consensus.cc:2781] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:14.902966 6734 raft_consensus.cc:686] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Becoming Leader. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Running, Role: LEADER
I20240502 08:00:14.903112 6734 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:14.905128 28254 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: term changed from 1 to 2, leader changed from 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) to 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199). New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:14.914585 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:14.916348 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:14.917446 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:14.927517 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:14.927922 7043 tablet_copy_client.cc:523] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:14.930004 7043 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:00:14.932615 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:14.943502 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:14.996150 7043 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:00:14.996760 30346 raft_consensus.cc:1203] T 57f7c358ace742649df52380939ece9d P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20240502 08:00:14.996968 7043 tablet_bootstrap.cc:492] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:00:14.997157 7043 ts_tablet_manager.cc:1367] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.067s user 0.029s sys 0.002s
I20240502 08:00:14.997846 7043 raft_consensus.cc:348] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 3 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:14.998118 7043 raft_consensus.cc:729] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:00:14.998239 7043 consensus_queue.cc:260] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 3.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:14.998525 7043 ts_tablet_manager.cc:1398] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:00:14.999735 30235 tablet_copy_service.cc:342] P 6c68c916f39a45f697df7eaf8973225a: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-5d1c1fa046b741aa8cf296aa57847fe3 received from {username='slave'} at 127.16.36.193:58623
I20240502 08:00:14.999832 30235 tablet_copy_service.cc:433] P 6c68c916f39a45f697df7eaf8973225a: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-5d1c1fa046b741aa8cf296aa57847fe3 on tablet 5d1c1fa046b741aa8cf296aa57847fe3 with peer 71922d97e1a4414698ecd14a7b2dedec
I20240502 08:00:15.012779 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.026732 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:15.027066 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:15.027328 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.045207 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.045277 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.047556 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.053337 30477 consensus_queue.cc:237] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 3.10, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:15.055012 28660 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:00:15.055544 7060 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:00:15.056561 30213 raft_consensus.cc:1261] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 3 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 08:00:15.057634 6734 consensus_queue.cc:1035] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:00:15.059396 6734 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 [term 3 LEADER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.061968 30213 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.062314 28257 catalog_manager.cc:5484] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 10 to 11, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 3 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.065871 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 04d358245cdf4b2ebb0b5005f75cf1e0 with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43174
I20240502 08:00:15.066917 7101 tablet_replica.cc:330] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:00:15.067677 7101 raft_consensus.cc:2229] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:15.067822 7101 raft_consensus.cc:2258] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:15.068511 7101 ts_tablet_manager.cc:1875] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:15.071288 7101 ts_tablet_manager.cc:1888] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.10
I20240502 08:00:15.071478 7101 log.cc:1196] T 04d358245cdf4b2ebb0b5005f75cf1e0 P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/04d358245cdf4b2ebb0b5005f75cf1e0
I20240502 08:00:15.072347 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 04d358245cdf4b2ebb0b5005f75cf1e0 (table rebalance_test_table_5 [id=ec76f09dd69640ff9c4a1023602ac6cd]) successfully deleted
I20240502 08:00:15.073415 28660 raft_consensus.cc:2932] T 04d358245cdf4b2ebb0b5005f75cf1e0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 3 FOLLOWER]: Committing config change with OpId 3.11: config changed from index 10 to 11, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
W20240502 08:00:15.095903 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.181836 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.185946 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:15.192847 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:15.206578 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.208699 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.229032 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.230181 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.240192 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:15.240372 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.250702 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.257848 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.260926 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.264961 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:15.275790 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.282056 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.293732 30213 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:15.294190 7060 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:15.297466 28398 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:15.298379 6734 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:15.335549 28528 raft_consensus.cc:1203] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:00:15.336400 7227 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:15.337069 28528 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:15.339829 28249 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:15.340524 30477 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:00:15.363669 28528 consensus_queue.cc:237] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:15.365628 28398 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:00:15.365733 30345 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 08:00:15.365913 28465 consensus_peers.cc:485] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): Couldn't send request to peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:15.366153 7261 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:15.366451 6826 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:15.368670 6826 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.369081 30345 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.371140 28398 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.371858 28254 catalog_manager.cc:5484] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: config changed from index 5 to 6, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.399003 28660 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:15.401032 30213 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240502 08:00:15.401140 28595 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:15.401402 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:15.401682 7291 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:00:15.435696 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:15.435984 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.439324 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.440660 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.444334 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.447300 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.454859 28398 raft_consensus.cc:1203] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Deduplicated request from leader. Original: 3.11->[3.12-3.12] Dedup: 3.12->[]
I20240502 08:00:15.456771 6367 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 LEADER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:15.457172 28398 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:15.458365 6367 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:58864 (request call id 0) took 1156 ms. Trace:
I20240502 08:00:15.458724 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:15.458438 6367 rpcz_store.cc:276] 0502 08:00:14.302288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 08:00:14.302376 (+ 88us) service_pool.cc:225] Handling call
0502 08:00:14.303584 (+ 1208us) log.cc:842] Serialized 489 byte log entry
0502 08:00:15.458350 (+1154766us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":60,"raft.queue_time_us":1607,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":307}
I20240502 08:00:15.461342 28256 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 3 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:15.461656 30346 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Committing config change with OpId 3.12: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:15.487250 7317 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5e9553dabc94f218f6427c1685ffc38)
I20240502 08:00:15.487459 7317 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:15.488535 7317 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189)
W20240502 08:00:15.490677 30148 leader_election.cc:334] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.492748 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "6c68c916f39a45f697df7eaf8973225a" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" is_pre_election: true
I20240502 08:00:15.493260 30148 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a; no voters: c39bcb719db947b8afb0bb141e1f7a4e, e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:15.493548 7317 raft_consensus.cc:2726] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20240502 08:00:15.550915 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.613816 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.616103 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.618360 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:15.618590 7260 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e5e9553dabc94f218f6427c1685ffc38)
I20240502 08:00:15.618707 7260 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:15.619210 7260 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:15.620757 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "6c68c916f39a45f697df7eaf8973225a" is_pre_election: true
I20240502 08:00:15.620915 30213 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 3.
I20240502 08:00:15.621217 28466 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
W20240502 08:00:15.621520 28463 leader_election.cc:334] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.621796 7264 raft_consensus.cc:2781] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20240502 08:00:15.621853 7264 raft_consensus.cc:482] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Starting leader election (detected failure of leader e5e9553dabc94f218f6427c1685ffc38)
I20240502 08:00:15.621913 7264 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:00:15.624691 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.625043 7264 raft_consensus.cc:504] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:15.625419 7264 leader_election.cc:288] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 4 election: Requested vote from peers e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407)
I20240502 08:00:15.627066 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.627535 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c65fe7f823df4b4996769fc784d3adb7" candidate_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:15.627691 30213 raft_consensus.cc:3037] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 3 FOLLOWER]: Advancing to term 4
I20240502 08:00:15.628803 30213 raft_consensus.cc:2454] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c39bcb719db947b8afb0bb141e1f7a4e in term 4.
I20240502 08:00:15.629133 28466 leader_election.cc:302] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, c39bcb719db947b8afb0bb141e1f7a4e; no voters:
I20240502 08:00:15.629348 7260 raft_consensus.cc:2781] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 4 FOLLOWER]: Leader election won for term 4
I20240502 08:00:15.629436 7260 raft_consensus.cc:686] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 4 LEADER]: Becoming Leader. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:15.629572 7260 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 3.9, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
W20240502 08:00:15.632262 28463 leader_election.cc:334] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.633141 28255 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: term changed from 3 to 4, leader changed from e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) to c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194). New cstate: current_term: 4 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:15.641893 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.642372 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.647619 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.663211 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.669140 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.676999 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.677632 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.693456 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.709836 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:15.730795 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:15.753459 30477 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:15.758476 30213 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:15.759514 7060 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:15.761597 6734 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:15.762061 30213 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:15.765098 28255 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 7 to 9, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:00:15.768255 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet a28d406b43194bc6b775707d021618f0 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:56398
I20240502 08:00:15.775796 7453 tablet_replica.cc:330] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:00:15.775970 7453 raft_consensus.cc:2229] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:15.776104 7453 raft_consensus.cc:2258] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:15.776233 30477 consensus_queue.cc:237] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:15.777024 7453 ts_tablet_manager.cc:1875] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:15.777910 30213 raft_consensus.cc:1261] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:00:15.778430 30658 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:58.457278 30664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:58.457626 30664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:58.464097 30664 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:59.521908 30664 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 08:00:00.126871 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.126984 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.127085 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.127135 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.127193 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.127329 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.128110 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.128197 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.128266 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.128316 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.128363 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.128402 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.128566 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.128636 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:00.250589 30664 rebalancer_tool.cc:1074] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:00.324033 30664 rebalancer_tool.cc:1074] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:00.477329 30664 rebalancer_tool.cc:1074] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.600621 30664 rebalancer_tool.cc:1074] tablet 3598dbda342f49aea2384b5beb17d7a1: 'de423605477a4819a9569821f59f03e4' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:00.777324 30664 rebalancer_tool.cc:1074] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:01.110224 30664 rebalancer_tool.cc:1074] tablet 07461c988fb3402c91cfd9235efbb9b1: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:01.296170 30664 rebalancer_tool.cc:1084] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6c68c916f39a45f697df7eaf8973225a' move ignored: Illegal state: tablet afc9c4d9dd5b4c039686ccd918616859: replica 6c68c916f39a45f697df7eaf8973225a is already present
I20240502 08:00:01.296269 30664 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:01.837792 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.837985 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.838063 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.838135 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.838214 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.838286 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.838368 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.839160 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.839417 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.839490 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.839537 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:01.912624 30664 rebalancer_tool.cc:1074] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:01.961736 30664 rebalancer_tool.cc:1084] tablet df20642aef7748ff8962b3095e1ab56c: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move ignored: Illegal state: tablet df20642aef7748ff8962b3095e1ab56c: replica 6c68c916f39a45f697df7eaf8973225a is already present
I20240502 08:00:01.961866 30664 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:02.590770 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.590984 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.591081 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.591156 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.591198 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.591261 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.591324 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.591369 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:02.621953 30664 rebalancer_tool.cc:1074] tablet 698f494a22d34e659766b97cb24d0293: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:02.630146 30664 rebalancer_tool.cc:1074] tablet 57f7c358ace742649df52380939ece9d: 'e5e9553dabc94f218f6427c1685ffc38' -> '' move scheduled
I20240502 08:00:02.651237 30664 rebalancer_tool.cc:1074] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:02.768435 30664 rebalancer_tool.cc:1074] tablet c65fe7f823df4b4996769fc784d3adb7: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:03.849171 30664 rebalancer_tool.cc:1136] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:07.491135 30664 rebalancer_tool.cc:1136] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:08.800516 30664 rebalancer_tool.cc:1136] tablet 07461c988fb3402c91cfd9235efbb9b1: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:08.820261 30664 rebalancer_tool.cc:1074] tablet 14bbfb3758e04aeab0df4bbab8380061: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:08.847025 30664 rebalancer_tool.cc:1074] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:08.925503 30664 rebalancer_tool.cc:1136] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:10.114151 30664 rebalancer_tool.cc:1136] tablet 57f7c358ace742649df52380939ece9d: 'e5e9553dabc94f218f6427c1685ffc38' -> '' move completed: OK
I20240502 08:00:10.122224 30664 rebalancer_tool.cc:1136] tablet 698f494a22d34e659766b97cb24d0293: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:10.134923 30664 rebalancer_tool.cc:1136] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:10.211055 30664 rebalancer_tool.cc:1136] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:10.242462 30664 rebalancer_tool.cc:1136] tablet 3598dbda342f49aea2384b5beb17d7a1: 'de423605477a4819a9569821f59f03e4' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:10.266853 30664 rebalancer_tool.cc:1136] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:10.278609 30664 rebalancer_tool.cc:1136] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:10.290371 30664 rebalancer_tool.cc:1074] tablet a28d406b43194bc6b775707d021618f0: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:10.303228 30664 rebalancer_tool.cc:1074] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:10.319397 30664 rebalancer_tool.cc:1074] tablet 5d1c1fa046b741aa8cf296aa57847fe3: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:10.362637 30664 rebalancer_tool.cc:1136] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:10.503479 30664 rebalancer_tool.cc:1136] tablet 14bbfb3758e04aeab0df4bbab8380061: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:10.518147 30664 rebalancer_tool.cc:1136] tablet 5d1c1fa046b741aa8cf296aa57847fe3: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:11.055181 30664 rebalancer_tool.cc:1136] tablet c65fe7f823df4b4996769fc784d3adb7: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:11.172592 30664 rebalancer_tool.cc:1074] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:11.226434 30664 rebalancer_tool.cc:1074] tablet 1631d3e38209497ea2454fdfc555a319: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:11.236863 30664 rebalancer_tool.cc:1074] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:11.851800 30664 rebalancer_tool.cc:1136] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:12.614363 30664 rebalancer_tool.cc:1136] tablet a28d406b43194bc6b775707d021618f0: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:12.624045 30664 rebalancer_tool.cc:1136] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:12.638890 30664 rebalancer_tool.cc:1136] tablet 1631d3e38209497ea2454fdfc555a319: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
W20240502 08:00:13.097186 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server 6c68c916f39a45f697df7eaf8973225a to server ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:00:13.143954 30664 rebalancer_tool.cc:1074] tablet 21d939e5035541a4b003c5d276493d5c: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:13.276106 30664 rebalancer_tool.cc:1074] tablet 9c7e7dadff0a4cd08deb043a37a54424: '6c68c916f39a45f697df7eaf8973225a' -> 'c39bcb719db947b8afb0bb141e1f7a4e' move scheduled
I20240502 08:00:13.302210 30664 rebalancer_tool.cc:1084] tablet ab785e07fa8248e5889bea8ac755d002: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move ignored: Illegal state: tablet ab785e07fa8248e5889bea8ac755d002: replica 29126d5aee924f81bdd47cc7153420a1 is already present
I20240502 08:00:13.302292 30664 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:13.709291 30664 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:13.859598 30664 rebalancer_tool.cc:1074] tablet 57f7c358ace742649df52380939ece9d: 'de423605477a4819a9569821f59f03e4' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:13.956396 30664 rebalancer_tool.cc:1074] tablet d08ad2a981fc4c13bc89d22feea7d556: '29126d5aee924f81bdd47cc7153420a1' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:14.069584 30664 rebalancer_tool.cc:1074] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: '6c68c916f39a45f697df7eaf8973225a' -> 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' move scheduled
I20240502 08:00:15.460119 30664 rebalancer_tool.cc:1074] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'de423605477a4819a9569821f59f03e4' -> '71922d97e1a4414698ecd14a7b2dedec' move scheduled
I20240502 08:00:15.509677 30664 rebalancer_tool.cc:1084] tablet 698f494a22d34e659766b97cb24d0293: 'e5e9553dabc94f218f6427c1685ffc38' -> 'c39bcb719db947b8afb0bb141e1f7a4e' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:15.509792 30664 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): unacceptable health status UNAVAILABLE
I20240502 08:00:15.778878 7060 consensus_queue.cc:1035] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:00:15.779506 30412 consensus_peers.cc:485] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 -> Peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): Couldn't send request to peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a28d406b43194bc6b775707d021618f0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:15.781013 7060 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.781940 7453 ts_tablet_manager.cc:1888] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 08:00:15.782025 7453 log.cc:1196] T a28d406b43194bc6b775707d021618f0 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/a28d406b43194bc6b775707d021618f0
I20240502 08:00:15.782852 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet a28d406b43194bc6b775707d021618f0 (table rebalance_test_table_8 [id=66e1eb2ffd3a40e499cba7cbdd607f68]) successfully deleted
I20240502 08:00:15.783813 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet a28d406b43194bc6b775707d021618f0 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:00:15.784433 28255 catalog_manager.cc:5484] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 9 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.784385 30213 raft_consensus.cc:2932] T a28d406b43194bc6b775707d021618f0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:15.818877 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:15.821171 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:15.823037 7483 ts_tablet_manager.cc:902] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:00:15.823622 7483 tablet_copy_client.cc:308] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:00:15.824134 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet 698f494a22d34e659766b97cb24d0293 from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:45489)
I20240502 08:00:15.824241 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet 698f494a22d34e659766b97cb24d0293 from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:45489: session id = 71922d97e1a4414698ecd14a7b2dedec-698f494a22d34e659766b97cb24d0293
I20240502 08:00:15.825320 28681 tablet_copy_source_session.cc:205] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:15.826478 7483 ts_tablet_manager.cc:1875] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:15.829262 7483 ts_tablet_manager.cc:1888] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:00:15.829502 7483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 698f494a22d34e659766b97cb24d0293. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:15.831004 7483 tablet_copy_client.cc:783] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:15.831166 7483 tablet_copy_client.cc:647] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:15.831285 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:15.831555 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.831876 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:15.832108 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.833884 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:15.863204 7483 tablet_copy_client.cc:523] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:15.865016 7483 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:00:15.890017 7491 ts_tablet_manager.cc:902] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:00:15.890589 7491 tablet_copy_client.cc:308] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:00:15.891125 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 167b5f3b818048af81b1350839ebe489 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 ({username='slave'} at 127.16.36.195:52847)
I20240502 08:00:15.891227 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 167b5f3b818048af81b1350839ebe489 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 at {username='slave'} at 127.16.36.195:52847: session id = ed3ca7e829bc42b5a8e9a7d1bdcc5a44-167b5f3b818048af81b1350839ebe489
I20240502 08:00:15.891541 7483 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:00:15.891820 7483 tablet_bootstrap.cc:492] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:00:15.892000 7483 ts_tablet_manager.cc:1367] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.027s user 0.015s sys 0.004s
I20240502 08:00:15.892060 28548 tablet_copy_source_session.cc:205] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:15.892436 7483 raft_consensus.cc:348] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:15.892746 7483 raft_consensus.cc:729] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:00:15.892856 7491 ts_tablet_manager.cc:1875] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:15.892863 7483 consensus_queue.cc:260] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:15.893142 7483 ts_tablet_manager.cc:1398] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 08:00:15.893672 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-698f494a22d34e659766b97cb24d0293 received from {username='slave'} at 127.16.36.193:45489
I20240502 08:00:15.893759 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-698f494a22d34e659766b97cb24d0293 on tablet 698f494a22d34e659766b97cb24d0293 with peer 71922d97e1a4414698ecd14a7b2dedec
I20240502 08:00:15.895418 7491 ts_tablet_manager.cc:1888] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:00:15.895653 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167b5f3b818048af81b1350839ebe489. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:15.896881 7491 tablet_copy_client.cc:783] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:15.897015 7491 tablet_copy_client.cc:647] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:15.898272 7491 tablet_copy_client.cc:523] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:15.899415 7491 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 08:00:15.920140 7491 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:15.920439 7491 tablet_bootstrap.cc:492] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap complete.
I20240502 08:00:15.920658 7491 ts_tablet_manager.cc:1367] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.003s
I20240502 08:00:15.921087 7491 raft_consensus.cc:348] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:15.921211 7491 raft_consensus.cc:729] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 08:00:15.921370 7491 consensus_queue.cc:260] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:15.921659 7491 ts_tablet_manager.cc:1398] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:00:15.922276 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-167b5f3b818048af81b1350839ebe489 received from {username='slave'} at 127.16.36.195:52847
I20240502 08:00:15.922353 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-167b5f3b818048af81b1350839ebe489 on tablet 167b5f3b818048af81b1350839ebe489 with peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:00:16.046949 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.048533 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.051178 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:16.056658 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:16.057168 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:16.060336 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:16.067392 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:16.084532 30213 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 4 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 3 index: 9. Preceding OpId from leader: term: 4 index: 10. (index mismatch)
I20240502 08:00:16.084981 7261 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240502 08:00:16.103216 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:00:16.114995 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:16.126080 30656 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:58.427073 30666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:58.427378 30666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:58.432087 30666 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:59.462850 30666 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:59.926360 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.926533 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.926592 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.926718 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.926757 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.926810 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.926856 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.926901 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.926949 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.926998 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.927050 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.927104 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.927155 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.927206 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.927256 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:00.135923 30666 rebalancer_tool.cc:1084] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:00:00.136066 30666 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:00.453322 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.453456 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.453512 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.453588 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.453635 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.453671 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.453706 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.453758 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.453789 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.453823 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.453888 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.453938 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.454044 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.454102 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.454154 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.454212 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.454243 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.454272 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:00.454303 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:00.454339 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:00.454385 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:00.454420 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:00.541796 30666 rebalancer_tool.cc:1074] tablet 57f7c358ace742649df52380939ece9d: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:00.637311 30666 rebalancer_tool.cc:1074] tablet afc9c4d9dd5b4c039686ccd918616859: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.907805 30666 rebalancer_tool.cc:1074] tablet f98044a1730e4a04a7088199d4f0850e: '71922d97e1a4414698ecd14a7b2dedec' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:01.226974 30666 rebalancer_tool.cc:1084] tablet 14bbfb3758e04aeab0df4bbab8380061: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240502 08:00:01.227067 30666 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:01.698135 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.698315 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.698395 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.698451 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.698498 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.698578 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.698629 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.698678 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.698730 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.698783 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.698835 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.698884 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.698928 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.698976 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.699018 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:01.768976 30666 rebalancer_tool.cc:1074] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:01.798204 30666 rebalancer_tool.cc:1074] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:01.931077 30666 rebalancer_tool.cc:1074] tablet df20642aef7748ff8962b3095e1ab56c: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:02.055083 30666 rebalancer_tool.cc:1074] tablet 6a40382745f84f0eadf9e0ee2e057289: '71922d97e1a4414698ecd14a7b2dedec' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:02.111279 30666 rebalancer_tool.cc:1074] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:02.149433 30666 rebalancer_tool.cc:1074] tablet 698f494a22d34e659766b97cb24d0293: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:02.321174 30666 rebalancer_tool.cc:1084] tablet dcd68a21ed7b4205a2f0cf75f80df0e6: '71922d97e1a4414698ecd14a7b2dedec' -> '6c68c916f39a45f697df7eaf8973225a' move ignored: Illegal state: tablet dcd68a21ed7b4205a2f0cf75f80df0e6: replica 6c68c916f39a45f697df7eaf8973225a is already present
I20240502 08:00:02.321281 30666 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:03.031533 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server e5e9553dabc94f218f6427c1685ffc38 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:03.031823 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:03.031935 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:03.032028 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:03.032078 30666 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:03.109632 30666 rebalancer_tool.cc:1074] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:05.181210 30666 rebalancer_tool.cc:1136] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:05.683534 30666 rebalancer_tool.cc:1136] tablet 57f7c358ace742649df52380939ece9d: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: Incomplete: tablet 57f7c358ace742649df52380939ece9d, TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> TS e5e9553dabc94f218f6427c1685ffc38 move failed, target replica disappeared
I20240502 08:00:06.320575 30666 rebalancer_tool.cc:1136] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:08.069308 30666 rebalancer_tool.cc:1074] tablet 21d939e5035541a4b003c5d276493d5c: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:08.111747 30666 rebalancer_tool.cc:1074] tablet 14bbfb3758e04aeab0df4bbab8380061: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:08.145994 30666 rebalancer_tool.cc:1136] tablet 14bbfb3758e04aeab0df4bbab8380061: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:08.781365 30666 rebalancer_tool.cc:1136] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:08.805472 30666 rebalancer_tool.cc:1136] tablet 6a40382745f84f0eadf9e0ee2e057289: '71922d97e1a4414698ecd14a7b2dedec' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:08.828048 30666 rebalancer_tool.cc:1136] tablet afc9c4d9dd5b4c039686ccd918616859: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:09.539415 30666 rebalancer_tool.cc:1136] tablet f98044a1730e4a04a7088199d4f0850e: '71922d97e1a4414698ecd14a7b2dedec' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:09.580196 30666 rebalancer_tool.cc:1136] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:09.599087 30666 rebalancer_tool.cc:1136] tablet df20642aef7748ff8962b3095e1ab56c: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:09.618790 30666 rebalancer_tool.cc:1074] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:09.629469 30666 rebalancer_tool.cc:1074] tablet ab785e07fa8248e5889bea8ac755d002: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:09.648124 30666 rebalancer_tool.cc:1074] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:10.660604 30666 rebalancer_tool.cc:1136] tablet ab785e07fa8248e5889bea8ac755d002: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:10.704732 30666 rebalancer_tool.cc:1136] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:10.812422 30666 rebalancer_tool.cc:1136] tablet 21d939e5035541a4b003c5d276493d5c: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:10.887771 30666 rebalancer_tool.cc:1136] tablet 698f494a22d34e659766b97cb24d0293: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:10.915468 30666 rebalancer_tool.cc:1074] tablet 07461c988fb3402c91cfd9235efbb9b1: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:10.940541 30666 rebalancer_tool.cc:1074] tablet 7dbf7921ae84476ab53fbd4707f463eb: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:10.951514 30666 rebalancer_tool.cc:1074] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:10.968832 30666 rebalancer_tool.cc:1074] tablet 3598dbda342f49aea2384b5beb17d7a1: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:11.836957 30666 rebalancer_tool.cc:1136] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:12.368569 30666 rebalancer_tool.cc:1136] tablet 07461c988fb3402c91cfd9235efbb9b1: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:13.054101 30666 rebalancer_tool.cc:1074] tablet 1631d3e38209497ea2454fdfc555a319: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:13.085727 30666 rebalancer_tool.cc:1074] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:13.548633 30666 rebalancer_tool.cc:1136] tablet 3598dbda342f49aea2384b5beb17d7a1: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:13.565690 30666 rebalancer_tool.cc:1136] tablet 7dbf7921ae84476ab53fbd4707f463eb: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:13.587829 30666 rebalancer_tool.cc:1136] tablet 1631d3e38209497ea2454fdfc555a319: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:13.674490 30666 rebalancer_tool.cc:1136] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:13.690342 30666 rebalancer_tool.cc:1074] tablet a28d406b43194bc6b775707d021618f0: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:13.767840 30666 rebalancer_tool.cc:1074] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:13.801864 30666 rebalancer_tool.cc:1074] tablet c65fe7f823df4b4996769fc784d3adb7: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:13.923148 30666 rebalancer_tool.cc:1136] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
W20240502 08:00:14.663985 6571 connection.cc:535] client connection to 127.16.36.200:43189 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:00:14.664419 30666 rebalancer_tool.cc:1123] tablet c65fe7f823df4b4996769fc784d3adb7: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move is abandoned: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:00:15.100602 30666 rebalancer_tool.cc:1136] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:15.974350 30666 rebalancer_tool.cc:1136] tablet a28d406b43194bc6b775707d021618f0: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
Illegal state: tablet server e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): unacceptable health status UNAVAILABLE
I20240502 08:00:16.154655 7491 ts_tablet_manager.cc:902] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Initiating tablet copy from peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589)
I20240502 08:00:16.155160 7491 tablet_copy_client.cc:308] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.199:43589
I20240502 08:00:16.155690 30497 tablet_copy_service.cc:140] P 29126d5aee924f81bdd47cc7153420a1: Received BeginTabletCopySession request for tablet a28d406b43194bc6b775707d021618f0 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 ({username='slave'} at 127.16.36.195:52903)
I20240502 08:00:16.155814 30497 tablet_copy_service.cc:161] P 29126d5aee924f81bdd47cc7153420a1: Beginning new tablet copy session on tablet a28d406b43194bc6b775707d021618f0 from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 at {username='slave'} at 127.16.36.195:52903: session id = ed3ca7e829bc42b5a8e9a7d1bdcc5a44-a28d406b43194bc6b775707d021618f0
I20240502 08:00:16.156828 30497 tablet_copy_source_session.cc:205] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:16.159679 7491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a28d406b43194bc6b775707d021618f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:16.163898 7491 tablet_copy_client.cc:783] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:16.164141 7491 tablet_copy_client.cc:647] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:16.166246 7491 tablet_copy_client.cc:523] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:16.167898 7491 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 08:00:16.180294 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.182478 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:16.187794 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:16.191035 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:16.195806 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:16.197628 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.201961 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:16.213805 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:16.235333 7491 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:00:16.235672 7491 tablet_bootstrap.cc:492] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap complete.
I20240502 08:00:16.235858 7491 ts_tablet_manager.cc:1367] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.000s
I20240502 08:00:16.236294 7491 raft_consensus.cc:348] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:16.236413 7491 raft_consensus.cc:729] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 08:00:16.236589 7491 consensus_queue.cc:260] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:16.236827 7491 ts_tablet_manager.cc:1398] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 08:00:16.236828 28398 raft_consensus.cc:1203] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240502 08:00:16.238085 6955 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:16.238396 28398 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:16.242478 28255 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:16.242776 30213 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:16.246685 30497 tablet_copy_service.cc:342] P 29126d5aee924f81bdd47cc7153420a1: Request end of tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-a28d406b43194bc6b775707d021618f0 received from {username='slave'} at 127.16.36.195:52903
I20240502 08:00:16.246781 30497 tablet_copy_service.cc:433] P 29126d5aee924f81bdd47cc7153420a1: ending tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-a28d406b43194bc6b775707d021618f0 on tablet a28d406b43194bc6b775707d021618f0 with peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:00:16.270553 28790 consensus_queue.cc:237] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
W20240502 08:00:16.272473 28726 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:16.272578 28528 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 08:00:16.272807 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:00:16.273061 7610 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:00:16.502244 28660 raft_consensus.cc:1203] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
W20240502 08:00:16.514322 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:16.543442 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.544075 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:16.549137 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:16.553297 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:16.554976 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:16.565474 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:16.566220 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:16.585970 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:00:16.590049 28660 raft_consensus.cc:1203] T a28d406b43194bc6b775707d021618f0 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240502 08:00:16.705132 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:16.755184 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:16.805452 7659 ts_tablet_manager.cc:902] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:16.806036 7659 tablet_copy_client.cc:308] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.196:33237
I20240502 08:00:16.806526 28811 tablet_copy_service.cc:140] P de423605477a4819a9569821f59f03e4: Received BeginTabletCopySession request for tablet fddc382ab3164ccdaa86f802c83da8be from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:39313)
I20240502 08:00:16.806629 28811 tablet_copy_service.cc:161] P de423605477a4819a9569821f59f03e4: Beginning new tablet copy session on tablet fddc382ab3164ccdaa86f802c83da8be from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:39313: session id = 71922d97e1a4414698ecd14a7b2dedec-fddc382ab3164ccdaa86f802c83da8be
I20240502 08:00:16.807536 28811 tablet_copy_source_session.cc:205] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:00:16.808307 7659 ts_tablet_manager.cc:1875] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:00:16.811004 7659 ts_tablet_manager.cc:1888] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240502 08:00:16.811213 7659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fddc382ab3164ccdaa86f802c83da8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:00:16.812377 7659 tablet_copy_client.cc:783] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 0 data blocks...
I20240502 08:00:16.812554 7659 tablet_copy_client.cc:647] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:00:16.814033 7659 tablet_copy_client.cc:523] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:00:16.815243 7659 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:00:16.834446 7659 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 08:00:16.834790 7659 tablet_bootstrap.cc:492] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:00:16.834964 7659 ts_tablet_manager.cc:1367] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240502 08:00:16.835376 7659 raft_consensus.cc:348] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } }
I20240502 08:00:16.835625 7659 raft_consensus.cc:729] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:00:16.835755 7659 consensus_queue.cc:260] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:00:16.836064 7659 ts_tablet_manager.cc:1398] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:00:16.836723 28811 tablet_copy_service.cc:342] P de423605477a4819a9569821f59f03e4: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-fddc382ab3164ccdaa86f802c83da8be received from {username='slave'} at 127.16.36.193:39313
I20240502 08:00:16.836863 28811 tablet_copy_service.cc:433] P de423605477a4819a9569821f59f03e4: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-fddc382ab3164ccdaa86f802c83da8be on tablet fddc382ab3164ccdaa86f802c83da8be with peer 71922d97e1a4414698ecd14a7b2dedec
W20240502 08:00:16.845541 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:16.854961 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:00:16.882107 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:16.883217 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:16.883541 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.886206 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:16.891196 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:16.894626 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:16.911895 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:16.960765 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:17.072144 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:17.128856 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:17.146621 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:17.185032 28398 raft_consensus.cc:1203] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240502 08:00:17.185827 7610 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:17.186414 28398 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:17.188962 28528 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:00:17.190382 28251 catalog_manager.cc:5484] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 5 to 6, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "e5e9553dabc94f218f6427c1685ffc38" member_type: VOTER last_known_addr { host: "127.16.36.200" port: 43189 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
W20240502 08:00:17.195799 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:17.226856 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:00:17.233170 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.234752 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:17.242182 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.249097 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:17.265623 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.265843 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.276135 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:17.332537 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:17.334498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.335131 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:17.339380 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.345161 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.348333 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:17.360215 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:17.370102 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:17.465848 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:17.500614 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:17.505211 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:17.508286 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.509557 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:17.534004 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.534005 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.541425 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:17.597410 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:17.601977 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:17.670213 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:17.689909 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:17.694744 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.698989 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:17.705818 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.716593 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:17.724335 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.735138 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.764379 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:17.768723 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.779505 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:17.789158 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.789932 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.801445 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:17.803277 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.903671 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.904182 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.907249 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:17.913390 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:17.916595 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:17.916615 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:17.917326 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:17.988922 28528 consensus_queue.cc:237] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:00:17.990336 28398 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:17.990365 28660 raft_consensus.cc:1261] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:00:17.990846 7261 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:17.991228 7261 consensus_queue.cc:1035] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:17.993644 7707 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:17.995239 28398 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:17.997287 28254 catalog_manager.cc:5484] T 167b5f3b818048af81b1350839ebe489 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 6 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:17.999486 28660 raft_consensus.cc:2932] T 167b5f3b818048af81b1350839ebe489 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:00:18.001629 30326 tablet_service.cc:1514] Processing DeleteTablet for tablet 167b5f3b818048af81b1350839ebe489 with delete_type TABLET_DATA_TOMBSTONED (TS 6efe17dcca7b4babad4785d19552b699 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33066
I20240502 08:00:18.004531 8010 tablet_replica.cc:330] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: stopping tablet replica
I20240502 08:00:18.004877 8010 raft_consensus.cc:2229] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:18.005071 8010 raft_consensus.cc:2258] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:18.005934 8010 ts_tablet_manager.cc:1875] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:18.008230 8010 ts_tablet_manager.cc:1888] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 08:00:18.008327 8010 log.cc:1196] T 167b5f3b818048af81b1350839ebe489 P 6efe17dcca7b4babad4785d19552b699: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-5/wal/wals/167b5f3b818048af81b1350839ebe489
I20240502 08:00:18.009627 28243 catalog_manager.cc:4830] TS 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): tablet 167b5f3b818048af81b1350839ebe489 (table rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]) successfully deleted
W20240502 08:00:18.022557 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:18.064638 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:00:18.092821 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.093914 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.097554 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.100803 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.103991 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.112725 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.124788 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.131395 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.146322 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.147097 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.148882 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.181921 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.184347 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.185353 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:18.206311 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:18.274739 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.280192 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.280718 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.283622 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.287659 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.287767 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.292654 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.406030 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.407171 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.408016 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.416157 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.418794 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.433892 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.436761 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:18.450152 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:18.528862 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:18.529908 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 08:00:18.530894 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.534838 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.535887 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.537839 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.558501 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.587442 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.626111 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.628371 28509 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.628875 30457 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.631065 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.632647 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.632939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.655849 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.689499 28528 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002"
dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:41206
I20240502 08:00:18.689677 28528 raft_consensus.cc:595] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Received request to transfer leadership
I20240502 08:00:18.738586 8243 raft_consensus.cc:979] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: : Instructing follower 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:18.738730 8243 raft_consensus.cc:1067] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Signalling peer 29126d5aee924f81bdd47cc7153420a1 to start an election
I20240502 08:00:18.739184 30477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002"
dest_uuid: "29126d5aee924f81bdd47cc7153420a1"
from {username='slave'} at 127.16.36.194:39689
I20240502 08:00:18.739326 30477 raft_consensus.cc:482] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:18.739387 30477 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:18.740582 30477 raft_consensus.cc:504] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:18.741065 30477 leader_election.cc:288] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 2 election: Requested vote from peers c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237), 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421)
I20240502 08:00:18.741621 28528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "c39bcb719db947b8afb0bb141e1f7a4e"
I20240502 08:00:18.741647 28790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 08:00:18.741752 28528 raft_consensus.cc:3032] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:18.741758 28790 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:18.741797 28528 raft_consensus.cc:729] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Running, Role: LEADER
I20240502 08:00:18.741959 28528 consensus_queue.cc:260] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:18.742141 28528 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:18.742937 28790 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:18.743203 28528 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:18.743739 30410 leader_election.cc:302] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 29126d5aee924f81bdd47cc7153420a1, c39bcb719db947b8afb0bb141e1f7a4e, de423605477a4819a9569821f59f03e4; no voters:
I20240502 08:00:18.743954 7802 raft_consensus.cc:2781] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:18.744136 7802 raft_consensus.cc:686] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Becoming Leader. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Running, Role: LEADER
I20240502 08:00:18.744313 30345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab785e07fa8248e5889bea8ac755d002" candidate_uuid: "29126d5aee924f81bdd47cc7153420a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "6efe17dcca7b4babad4785d19552b699"
I20240502 08:00:18.744292 7802 consensus_queue.cc:237] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:18.744424 30345 raft_consensus.cc:3037] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:18.745497 30345 raft_consensus.cc:2454] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29126d5aee924f81bdd47cc7153420a1 in term 2.
I20240502 08:00:18.746977 28252 catalog_manager.cc:5484] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: term changed from 1 to 2, leader changed from c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) to 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199). New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:18.762914 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.763832 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:18.771142 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.775274 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.777869 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.787158 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.807739 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:18.825368 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:18.867655 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 08:00:18.883148 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:18.915722 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:18.921701 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:18.922148 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:18.923777 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:18.947674 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.950580 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:18.960399 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.978646 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:18.988909 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:18.991278 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:18.996644 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:18.998400 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:19.012646 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.031245 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.114073 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.118669 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:19.120724 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.121300 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.124280 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.133014 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:19.141819 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.164772 28528 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:19.165690 8299 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:19.180577 28790 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:19.181210 8299 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:00:19.184819 30345 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:00:19.185353 8440 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 08:00:19.198215 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:00:19.227885 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:19.265120 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:19.267192 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.270658 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.270877 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:19.288717 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.293119 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:19.295740 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.307551 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:19.326810 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:19.334507 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.341946 28660 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859"
dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44794
I20240502 08:00:19.342109 28660 raft_consensus.cc:595] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Received request to transfer leadership
W20240502 08:00:19.362350 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.368917 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.371410 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.378120 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.379099 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:19.381429 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.397430 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.398579 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240502 08:00:19.416792 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.465180 30326 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:00:19.466884 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.471428 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.471570 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.477365 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.483292 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.485538 28639 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 08:00:19.510478 8528 raft_consensus.cc:979] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: : Instructing follower 6efe17dcca7b4babad4785d19552b699 to start an election
I20240502 08:00:19.510600 8528 raft_consensus.cc:1067] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Signalling peer 6efe17dcca7b4babad4785d19552b699 to start an election
I20240502 08:00:19.511148 30345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859"
dest_uuid: "6efe17dcca7b4babad4785d19552b699"
from {username='slave'} at 127.16.36.195:57097
I20240502 08:00:19.511299 30345 raft_consensus.cc:482] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:00:19.511353 30345 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:19.512507 30345 raft_consensus.cc:504] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:19.513116 30345 leader_election.cc:288] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [CANDIDATE]: Term 2 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407), de423605477a4819a9569821f59f03e4 (127.16.36.196:33237)
I20240502 08:00:19.513417 28660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "6efe17dcca7b4babad4785d19552b699" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 08:00:19.513551 28660 raft_consensus.cc:3032] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:00:19.513620 28660 raft_consensus.cc:729] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 08:00:19.513695 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "6efe17dcca7b4babad4785d19552b699" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6c68c916f39a45f697df7eaf8973225a"
I20240502 08:00:19.513830 30215 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:19.513773 28660 consensus_queue.cc:260] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:19.513974 28660 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:00:19.514765 28660 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efe17dcca7b4babad4785d19552b699 in term 2.
I20240502 08:00:19.514917 30215 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efe17dcca7b4babad4785d19552b699 in term 2.
I20240502 08:00:19.515300 30282 leader_election.cc:302] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6c68c916f39a45f697df7eaf8973225a, 6efe17dcca7b4babad4785d19552b699, ed3ca7e829bc42b5a8e9a7d1bdcc5a44; no voters:
I20240502 08:00:19.515365 28790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afc9c4d9dd5b4c039686ccd918616859" candidate_uuid: "6efe17dcca7b4babad4785d19552b699" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "de423605477a4819a9569821f59f03e4"
I20240502 08:00:19.515508 8445 raft_consensus.cc:2781] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:00:19.515515 28790 raft_consensus.cc:3037] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Advancing to term 2
W20240502 08:00:19.515551 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.515586 8445 raft_consensus.cc:686] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 2 LEADER]: Becoming Leader. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Running, Role: LEADER
I20240502 08:00:19.515741 8445 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:19.516733 28790 raft_consensus.cc:2454] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efe17dcca7b4babad4785d19552b699 in term 2.
I20240502 08:00:19.518046 28251 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: term changed from 1 to 2, leader changed from ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) to 6efe17dcca7b4babad4785d19552b699 (127.16.36.198). New cstate: current_term: 2 leader_uuid: "6efe17dcca7b4babad4785d19552b699" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
W20240502 08:00:19.518874 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.616685 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.617360 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.618062 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.618075 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.622675 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.627458 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.637126 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:19.671661 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:00:19.744048 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.747747 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:19.757931 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:00:19.758708 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:19.759181 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.763567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:19.766124 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240502 08:00:19.774668 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 11 similar messages]
I20240502 08:00:19.790578 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.795306 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.796059 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.813663 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.815408 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.819731 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.825268 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.828980 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.830513 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:19.842506 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:19.854272 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:19.866581 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240502 08:00:19.889490 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:19.952493 30215 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:19.952984 6240 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:19.956786 28660 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:19.957217 6240 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:19.961227 28791 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:00:19.962652 8445 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:00:19.968303 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:19.974638 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:19.976439 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:19.976742 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:19.977106 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:19.989176 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.003604 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:20.059001 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:20.117305 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.121106 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.130129 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.130394 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.131783 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.144325 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.155992 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:20.191952 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:00:20.192968 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:20.217510 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.218412 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.226177 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.236141 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.236289 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.247797 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.249971 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.310135 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:20.313373 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:20.316844 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.323208 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.326341 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.327100 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.327826 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.327972 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:20.375675 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:20.450424 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.456357 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.459293 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.459937 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.465907 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.466701 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.484858 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.598168 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.599181 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.600183 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.604200 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.608805 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.629170 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.636827 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.656816 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.660241 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.666810 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.668534 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.677002 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.680169 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.684667 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:20.757807 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:20.802136 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.802709 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.802819 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.806387 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.806680 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.815367 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.843892 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:20.890157 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:20.901518 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:20.953455 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:20.954962 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.974268 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:20.978850 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.985967 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:20.986963 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:20.987008 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:20.992931 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.006708 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.019824 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.033492 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:21.047139 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.048969 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.055532 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:21.092888 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:21.102682 30476 consensus_queue.cc:237] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:00:21.104024 30345 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:21.104183 28791 raft_consensus.cc:1261] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 29126d5aee924f81bdd47cc7153420a1: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:00:21.104408 9193 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:21.104669 8730 consensus_queue.cc:1035] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:00:21.106401 8730 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P 29126d5aee924f81bdd47cc7153420a1 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:21.107008 28791 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:21.108175 30345 raft_consensus.cc:2932] T ab785e07fa8248e5889bea8ac755d002 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:21.109767 28252 catalog_manager.cc:5484] T ab785e07fa8248e5889bea8ac755d002 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 6 to 8, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "29126d5aee924f81bdd47cc7153420a1" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:00:21.114521 28509 tablet_service.cc:1514] Processing DeleteTablet for tablet ab785e07fa8248e5889bea8ac755d002 with delete_type TABLET_DATA_TOMBSTONED (TS c39bcb719db947b8afb0bb141e1f7a4e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39626
I20240502 08:00:21.114938 9206 tablet_replica.cc:330] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: stopping tablet replica
I20240502 08:00:21.115099 9206 raft_consensus.cc:2229] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:21.115257 9206 raft_consensus.cc:2258] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:21.115837 9206 ts_tablet_manager.cc:1875] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:21.118350 9206 ts_tablet_manager.cc:1888] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:00:21.118427 9206 log.cc:1196] T ab785e07fa8248e5889bea8ac755d002 P c39bcb719db947b8afb0bb141e1f7a4e: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/ab785e07fa8248e5889bea8ac755d002
I20240502 08:00:21.118973 28241 catalog_manager.cc:4830] TS c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): tablet ab785e07fa8248e5889bea8ac755d002 (table rebalance_test_table_0 [id=f22192bf738e4f63a739dd5907881674]) successfully deleted
I20240502 08:00:21.188064 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.190521 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.191908 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.192081 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.195700 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.208856 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:21.210103 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:21.288316 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240502 08:00:21.315789 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.315909 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.319357 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.321274 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:21.325603 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:21.336275 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.342792 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:21.343466 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:21.347841 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:21.426277 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.427719 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.432078 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.437415 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.446692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:21.449837 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:21.463496 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.513075 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.513252 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.517244 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.521787 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.525426 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:21.549058 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.551795 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:21.602195 30345 consensus_queue.cc:237] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:00:21.603652 28791 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:00:21.603749 30215 raft_consensus.cc:1261] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer 6efe17dcca7b4babad4785d19552b699: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:00:21.604300 9389 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:00:21.604555 8776 consensus_queue.cc:1035] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:00:21.606683 9389 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:21.607120 30215 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:21.609961 28255 catalog_manager.cc:5484] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 reported cstate change: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "6efe17dcca7b4babad4785d19552b699" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:21.610139 28791 raft_consensus.cc:2932] T afc9c4d9dd5b4c039686ccd918616859 P de423605477a4819a9569821f59f03e4 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:00:21.615398 28639 tablet_service.cc:1514] Processing DeleteTablet for tablet afc9c4d9dd5b4c039686ccd918616859 with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:54006
I20240502 08:00:21.615810 9399 tablet_replica.cc:330] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:00:21.615974 9399 raft_consensus.cc:2229] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:00:21.616107 9399 raft_consensus.cc:2258] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:00:21.616735 9399 ts_tablet_manager.cc:1875] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:00:21.619207 9399 ts_tablet_manager.cc:1888] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:00:21.619432 9399 log.cc:1196] T afc9c4d9dd5b4c039686ccd918616859 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/afc9c4d9dd5b4c039686ccd918616859
I20240502 08:00:21.619982 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet afc9c4d9dd5b4c039686ccd918616859 (table rebalance_test_table_9 [id=172eddf6e26644619807f0650e92ccb4]) successfully deleted
W20240502 08:00:21.627995 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:21.689011 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.690418 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.691790 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.693845 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:21.694739 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:21.708505 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.726902 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:21.734323 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:21.768417 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:21.806991 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:21.819125 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.819922 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.831486 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.832175 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.834940 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:21.837342 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:21.866081 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.904446 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:21.905826 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.917290 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:21.919947 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:00:21.920208 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:21.920619 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:21.921769 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:21.925218 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:21.930539 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:21.944171 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:00:22.001467 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:00:22.007772 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:22.040575 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.045349 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.049466 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.050413 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.052556 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.065838 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.067724 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.175374 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.181278 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.201248 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.206707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.207132 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.226949 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.227284 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.236867 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.237049 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.246174 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.252521 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.253459 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.253911 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.266047 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:22.323033 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:00:22.357780 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:22.373859 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:22.387485 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.392048 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.393079 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.394187 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.395102 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.405758 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.413832 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:22.469790 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:22.529495 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.532562 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.532814 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.540694 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.555752 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.559244 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.559914 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.639614 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.640693 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.642657 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.649947 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.651865 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:22.660506 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:22.667019 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.677644 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:22.678928 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:22.735267 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.740613 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.742722 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.743163 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.747735 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.751719 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:22.789033 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:22.802356 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:22.880474 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:22.882426 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.882790 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.887387 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:22.891037 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:22.899466 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:22.904692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:22.959859 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:23.005901 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.010107 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.014715 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.025384 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.027135 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.035243 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.043027 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.099854 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.102674 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.103631 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.115016 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.115521 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.116698 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.129601 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:23.182075 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:23.229781 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.236138 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.236660 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.244843 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.246243 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.248167 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.254487 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.373162 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.374184 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.381867 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.391961 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.391997 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.402616 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.410825 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.435694 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.436198 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:23.441505 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:23.443348 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:23.443989 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.446175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.449935 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.464720 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:23.470769 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 08:00:23.472946 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.570516 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.577973 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.580889 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.581279 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.588830 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.598317 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.612926 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:23.678074 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240502 08:00:23.680095 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:23.714776 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.730154 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.730557 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.731869 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.732333 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.732488 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.749810 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.802168 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.808835 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.809939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.811647 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:23.827718 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.833472 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.847218 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.917977 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.918037 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:23.918594 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:23.922744 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:23.927671 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:23.942623 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:23.951881 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:23.955264 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.053434 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.054071 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:24.062500 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.070089 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.073750 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.077481 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.086262 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:24.155215 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:24.175019 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.175526 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:24.180999 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.186914 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.187095 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:24.209373 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:24.210227 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.214854 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.253474 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:24.257592 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.268885 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.269503 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.270208 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.279742 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.294088 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:24.376618 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:24.390128 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.393934 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.395602 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.402418 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.406608 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:24.414999 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:24.418581 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.438331 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:24.531011 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:24.543884 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:24.545563 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.547259 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.555801 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:24.559994 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.571668 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.571939 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.572188 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.605566 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.609607 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.617364 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.617743 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.628445 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:24.633881 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:00:24.643453 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:24.646910 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.650939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:24.653192 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:00:24.668471 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:24.743414 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.745693 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:24.755538 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.764568 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.769213 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.769690 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:24.773538 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:24.795337 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:00:24.820478 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:00:24.828855 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:00:24.829653 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:24.876752 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.877193 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:24.878518 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:24.883587 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:24.889187 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.899672 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:24.902357 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:24.930166 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:24.952600 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:00:24.954341 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:25.008015 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.009266 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.021111 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:25.021957 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.052004 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:25.055676 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:00:25.062178 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.062541 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.084806 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:25.097353 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.097882 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.098204 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.098654 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.100803 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.124702 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:25.135931 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:25.228621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.228709 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:25.228363 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:25.232069 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.232228 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.238865 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.239960 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.252314 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:25.361169 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:00:25.368209 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.368270 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.375620 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.379449 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.380482 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:25.382333 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:25.384593 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.421838 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:25.475868 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:25.567453 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.567817 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.568197 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.571089 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.573894 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.597819 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.599457 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:25.700910 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.706370 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.707756 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:25.710390 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.710970 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.716079 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.743657 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.781945 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:25.787652 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.793162 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.793551 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.801225 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.806279 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.820571 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:25.868433 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240502 08:00:25.898151 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:25.900575 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:25.917470 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.918154 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.918665 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:25.918663 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:25.923203 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:25.932787 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:25.956929 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:26.008891 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:26.054155 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.057425 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:26.058965 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.059307 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.060195 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.067760 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.068053 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.196170 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.196550 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.197276 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.202162 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.206437 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.227392 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:26.243705 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.246769 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.252701 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.255645 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.259277 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.270961 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:26.274448 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240502 08:00:26.286226 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.302275 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:26.400727 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.401561 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.413956 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.415551 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.415942 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.430575 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.430805 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:26.475878 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:26.497583 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:26.529412 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.534915 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.538887 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.541018 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.544577 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.553167 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.559513 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:26.583606 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:26.673746 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.685627 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.689625 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.691576 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.694075 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:26.697710 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.704700 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.713429 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.714069 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.724179 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.725422 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:26.729781 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.742828 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.752630 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:26.867255 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.870669 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:26.871484 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:26.874022 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:26.881414 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:26.887518 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:26.893774 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:26.898567 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:26.907632 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:00:26.946673 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:26.999420 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:27.004184 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.005954 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.010531 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.012909 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.017891 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.032114 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.044817 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:27.082242 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:00:27.085099 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.115972 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.115972 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.121176 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.131297 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.132193 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.135071 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.152755 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:27.170393 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:00:27.273109 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.347834 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.349013 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:27.352032 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.353647 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.353683 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.354117 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.354800 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.362653 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:27.391870 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:27.412909 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.477262 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.477298 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.479085 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.480650 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:27.482409 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.485365 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.505892 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:27.547539 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:27.553570 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.557055 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.563061 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.568141 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.569172 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.575065 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.605854 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:27.697774 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.698704 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.700696 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:27.703516 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.706667 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.707535 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.713284 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.740912 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:27.762379 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:27.835863 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.836689 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.847965 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:27.852083 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.857204 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:27.858860 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.874653 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:27.876355 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:27.909760 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.912572 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:27.921691 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:27.931480 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:27.933211 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:27.936717 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.942888 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:27.945109 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:28.049018 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.049962 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.057384 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.057404 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.063692 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:28.068823 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.070345 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.188371 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.195113 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.198773 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.200376 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.209760 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.223284 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:28.227569 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.274096 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.275516 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.281528 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.288182 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.293005 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.298135 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:28.304661 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.401782 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.404392 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.409956 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.412341 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.418570 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:28.426210 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:28.429587 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.429736 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.523741 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.525835 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.530988 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.535410 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.550225 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.550587 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:28.550788 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240502 08:00:28.555308 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:28.561280 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:28.623102 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:28.650365 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.660657 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.661154 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.662241 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:28.667732 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.670187 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.678146 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.737098 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.737352 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.738361 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.745386 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.767593 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.768630 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:28.778401 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:28.807104 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:28.818940 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240502 08:00:28.871174 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:28.877125 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:28.877983 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.880575 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:28.880832 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.882196 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:28.882643 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.001142 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.001992 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.004935 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.005971 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.009613 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.023851 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.024477 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:29.073637 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:29.224246 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:29.225281 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.227371 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.240657 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.241551 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.242399 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.245687 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.246320 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:29.316370 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:29.338371 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:29.357509 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:29.367249 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.367499 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.376904 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.381229 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.382089 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.392411 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.395591 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.401024 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.405879 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.422151 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.427217 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.427965 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.445324 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.459154 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:29.557638 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:29.560776 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.561892 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.565773 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.566466 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.572233 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.584960 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.595485 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:29.612852 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:29.635092 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:29.696185 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.699400 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.712814 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.713827 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.715305 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.715500 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.729897 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:29.738755 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:29.801276 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:00:29.801976 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:29.807720 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:29.820171 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:29.820580 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.823693 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.825604 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.831272 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.832074 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.836376 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:29.880808 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:00:29.881556 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:29.899670 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:29.902781 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:00:29.907253 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:29.907738 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.908835 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:29.911290 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:29.915161 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:29.936923 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.037439 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.037734 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.040581 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.042994 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:30.050884 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.051718 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.060992 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:30.090534 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:30.090951 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:30.184103 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.187431 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.194582 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.196666 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:30.214138 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.237093 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.243710 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.250332 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.262197 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.264020 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.266223 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.282457 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:30.286949 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:30.290633 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.290874 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:30.366418 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:30.390347 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:30.390973 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:00:30.392508 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.402417 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.413272 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.414927 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.416066 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.421056 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:30.434058 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:30.523260 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:00:30.528683 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:30.529194 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.530710 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.539732 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.540632 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.548738 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.549667 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.634217 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.644230 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.646332 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:30.648370 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:30.649813 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.660810 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.662500 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.683079 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.729141 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.731057 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.732769 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:30.743436 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:30.745730 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.747001 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:30.781343 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:30.783179 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.988066 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.988397 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:30.989653 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:30.991508 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:30.999513 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:31.004221 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.006134 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:31.067320 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:31.074769 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:31.110949 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.113297 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.115108 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.117838 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.121506 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.128202 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.140744 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:31.205659 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240502 08:00:31.223194 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:31.243741 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.244377 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.250808 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.254707 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.257251 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:31.271158 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.271441 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:31.303893 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240502 08:00:31.369707 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.376087 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.387049 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.393353 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.410485 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:31.421392 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.425676 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.428128 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.428620 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.429630 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.445271 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.445529 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:31.463155 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.477237 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:31.548097 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:31.578464 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.582093 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.587574 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.595788 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.597087 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.602908 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:31.608376 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:31.622608 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:31.659927 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:31.717183 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.721874 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.724498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.724627 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.745730 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:31.758716 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.764678 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:31.822347 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:31.827374 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.829798 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.835115 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.837363 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.839625 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:31.856382 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:31.861109 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.862481 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.927277 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:31.932621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.940284 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:31.942795 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:31.943128 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.946658 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:31.954805 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:31.961123 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240502 08:00:31.965811 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:32.065120 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.068605 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.068650 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.078011 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:32.079054 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.085696 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:32.088641 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:32.129388 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:32.182657 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.183660 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.187216 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.195438 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:32.216300 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:32.218137 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:32.226054 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.230293 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:32.252866 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:32.264137 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.264225 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.273636 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:32.274024 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:32.292135 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.295606 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.300674 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:32.343750 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:32.408339 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.410009 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:32.420200 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.421530 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.423480 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.423915 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:32.425560 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:32.437343 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:32.537698 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.538990 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.550349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:32.560382 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.560639 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:32.582842 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.588091 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:32.626736 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:32.634395 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:32.646560 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.648224 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.649077 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:32.656179 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:32.657451 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.668136 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.670807 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:32.694720 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:32.893376 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.894001 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:32.896044 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:32.900745 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.901434 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:32.908227 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:32.918104 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.028816 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:33.029284 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:33.029821 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.032225 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:33.042007 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:33.046617 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:33.054000 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:33.054391 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.076980 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.077289 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.106791 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.107683 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.112367 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.114686 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.124245 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.135322 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:33.135355 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:33.136639 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.247275 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.251497 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.252203 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.253307 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.256650 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.267035 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.281132 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.386237 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.387560 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.387775 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.392560 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.411460 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.412041 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.415035 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.474984 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.478014 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.479175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:33.485071 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:33.491024 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.504685 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.510833 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.512570 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:33.539057 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:33.578560 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.582237 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.583385 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.587354 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:33.588719 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:33.593022 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.595384 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.628396 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.721462 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.721940 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.722461 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.725708 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.726162 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.748023 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.769237 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:33.771471 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:33.859264 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.861429 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.862030 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.868660 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.871685 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.884608 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:33.886554 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:33.921196 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:33.924264 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240502 08:00:33.937947 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.941174 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:33.945454 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:33.949023 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:33.951437 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:33.961346 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:33.965708 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:33.973163 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:34.043108 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.082702 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.086589 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:34.087416 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.088495 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.089908 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.104049 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:34.108870 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.117758 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:34.125073 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.229588 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.239727 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.241649 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.249938 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.253091 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.255390 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:34.259655 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:34.278995 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.296129 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:34.298053 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.299266 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.306814 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:34.317488 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.323520 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:34.357370 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:00:34.389065 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:34.398926 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.424358 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.424865 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.428197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.437683 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.442099 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:34.444839 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.448807 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:34.524085 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:34.609057 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.609496 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.609778 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.614542 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:34.615623 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.618867 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.639526 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:34.640554 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.761334 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.764653 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:34.766625 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:34.766981 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.770136 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.781751 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.782967 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.808558 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:34.851706 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:00:34.905508 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:00:34.906522 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:34.908962 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:34.910445 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:34.911984 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:34.917912 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:00:34.919064 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:34.921223 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:34.930749 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:34.931588 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:34.948865 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:35.022392 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.027880 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.033205 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.035411 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.035768 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.053388 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.066377 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:35.075610 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:35.116341 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.116891 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.128669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:35.136096 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.138774 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.141111 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.142462 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:35.149049 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:35.254751 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.258868 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.262790 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.263104 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:35.270995 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.276120 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.285519 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:35.393332 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:35.394021 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.405514 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.410040 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.415067 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.429443 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.433130 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:35.455817 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.456097 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.458563 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.459416 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.463647 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.473212 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:35.489866 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:35.492276 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.507021 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:35.603679 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:35.610706 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.612280 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.613123 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.613415 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.613453 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:35.614722 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.617130 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:35.668099 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:35.724167 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:00:35.737605 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.739045 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.744235 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:35.749306 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.755777 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.759562 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.760605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:35.790017 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:00:35.863801 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.868335 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.870430 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.870944 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:35.878475 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.905498 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.915169 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.927048 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:35.932943 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.935647 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:35.940320 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:35.943358 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:35.944347 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:35.955817 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:35.956739 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.070896 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.073200 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:36.074450 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.079505 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.082108 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.084661 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.093706 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:36.120416 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:00:36.174165 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:36.200508 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.206605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.210261 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.212510 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.215416 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.220664 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.220892 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:36.278973 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240502 08:00:36.303839 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.304446 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.304541 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.313028 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.314391 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.316386 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:36.334491 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:36.342564 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:36.378870 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:36.487542 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240502 08:00:36.492619 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:36.550552 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.552507 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.553222 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.553804 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:36.567782 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.576673 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.578074 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:36.593879 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:36.692569 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.694083 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.695436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.705456 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:36.705753 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.712684 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.725714 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.769337 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.772265 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.776108 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.776432 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.779604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.782886 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.789672 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:36.803516 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:36.813295 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:36.870008 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:36.905931 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:36.906834 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:36.907720 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.908789 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:36.921173 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:36.926765 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:36.928637 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:36.954839 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:37.065078 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.067974 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.070789 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.083393 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.083730 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.087842 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.092555 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.095620 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:37.096568 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:37.105556 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.124213 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.124791 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:37.126842 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:37.135809 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.136351 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.153095 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:37.220906 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:37.257427 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.259114 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.265122 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.267467 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.273366 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.276005 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.285439 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.396113 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.396785 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.398617 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.400635 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.409504 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.409660 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.412827 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:37.471942 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:37.529034 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.531314 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.539152 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.543699 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.544023 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:37.546072 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:37.571275 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:37.576040 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.582083 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.589017 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.603685 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.604750 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.609418 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.622936 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.624106 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.632328 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:37.698834 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:37.730729 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.731946 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.734179 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.737455 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.741803 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.742722 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.772501 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.876062 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.878963 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.879225 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.886204 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.899432 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:37.926218 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.927129 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.964241 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:37.965474 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.966455 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:37.976321 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:37.980340 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:37.983975 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:37.989953 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:38.015214 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:38.016175 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.093456 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.093544 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:38.095360 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.099498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.102093 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.111855 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.130472 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:38.180971 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:38.196791 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:38.220134 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:38.312073 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.312398 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:38.317195 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.322721 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.323181 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.340682 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.348568 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.430025 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.430274 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.433156 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.436992 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:38.440637 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.441589 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.460346 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:38.554428 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:38.555450 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:38.559763 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:38.561767 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.564841 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.567229 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.570901 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.577217 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.579120 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.682967 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:38.685377 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.685693 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:38.692515 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240502 08:00:38.696746 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.699175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.701738 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.712563 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.770320 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.772753 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:38.777722 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.778404 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.779785 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.780196 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.803741 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:38.889158 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:38.901494 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.902598 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:38.904649 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:38.905158 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:38.917060 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:38.928673 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:38.944753 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:38.974212 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:00:39.025163 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:39.029139 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.030308 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.034634 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.039539 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.042104 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.044024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:39.056259 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:39.085556 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240502 08:00:39.099170 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:39.115864 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:39.116717 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:39.118074 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.118214 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.124650 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.129249 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.133989 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.156338 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:39.156400 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:39.270648 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.274385 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.280752 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.282537 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.302162 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.308137 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.316049 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:39.452363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:39.464095 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.473632 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.480656 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.500747 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.501533 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.513039 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.519691 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.522644 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.533011 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.534090 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:39.548671 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.548810 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.557991 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:39.583067 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:00:39.638383 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:39.666316 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.671027 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.676635 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.678418 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.680802 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:39.682324 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.693836 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:39.694906 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:39.710122 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:39.801957 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.808030 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.809243 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.821219 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.838019 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.841480 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.841804 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:39.885619 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:00:39.886687 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:39.901110 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:39.901969 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.905417 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:39.910643 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:39.929751 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:39.938482 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:00:39.942883 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:39.942919 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:39.977418 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:00:39.997866 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.010094 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.010532 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.014667 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.016927 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.023100 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.042941 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:40.059142 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:00:40.073768 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:40.136696 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:00:40.213271 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:40.250506 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.253021 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.253302 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.257498 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.265645 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.268286 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.283938 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.368144 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.368914 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.376147 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.382633 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.382798 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.391430 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.397866 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:40.418076 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:40.506415 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.507900 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.509457 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.514894 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.520738 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.520774 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.545596 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:40.557958 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:40.642097 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.652410 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.659456 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.663594 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:40.667630 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:40.682600 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.686479 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.687655 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.695338 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.711306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.712090 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.715071 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.716403 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.719244 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.723414 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:40.782593 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240502 08:00:40.793880 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:40.853134 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.858968 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.860911 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.861479 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.870386 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:40.871860 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:40.885183 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:40.992343 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:40.992602 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:40.998980 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.001895 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.019989 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:41.022848 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:00:41.035859 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.042145 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.103997 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.104809 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.106076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.106369 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.121837 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.122241 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:41.132594 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:41.132972 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:00:41.143831 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.206754 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.218415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.219321 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.223088 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.228319 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.234180 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.241674 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:41.262859 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:41.341697 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.342871 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.348696 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.351758 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.363619 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.363853 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:41.372334 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:41.377661 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:41.403323 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240502 08:00:41.469306 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.471940 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.483767 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.486579 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.487546 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:41.489010 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:41.490676 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.515633 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:41.525583 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:41.535779 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.543653 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.547034 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.552719 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.571866 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.582008 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.586405 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:41.679020 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:41.696131 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.696640 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.696897 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.705921 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.707643 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.712965 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.725593 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:41.785364 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:00:41.807188 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:00:41.814194 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:41.835538 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.843438 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.848225 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.852705 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.854229 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:41.858124 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.858595 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.925127 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.928974 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:41.934026 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:41.936828 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.941915 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:41.946079 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:41.967170 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:42.073015 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:00:42.115912 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:42.172155 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.173664 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.181372 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.182376 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.188964 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.190986 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.218915 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:42.225530 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:42.229821 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:42.308635 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.311138 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.315274 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.320611 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:42.347180 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.357575 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.359799 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.369035 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.380734 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.385986 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.388696 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.393862 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:42.402689 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:42.406183 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:42.406875 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:42.455984 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:00:42.505291 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:42.521850 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.522697 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.525455 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:42.533661 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.541082 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.543334 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.543998 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.662397 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.668320 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.673302 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.681270 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.682018 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.684994 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.692548 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:42.731725 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:42.757761 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:42.781570 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.794512 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.796077 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.796291 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.801614 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:42.805035 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.812163 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.865605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:42.866190 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:42.871742 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:42.874766 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:42.875294 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240502 08:00:42.879462 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:42.879913 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.880100 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:42.889060 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:42.998725 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.000612 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:43.001298 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.013432 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.019402 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:43.024549 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.045928 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:43.149224 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.149286 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:43.178536 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.178654 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.196098 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.196305 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:43.197532 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.206632 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:43.211452 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:43.212323 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.213132 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.222975 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:43.225749 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:43.240612 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:43.250016 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:43.253643 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:43.258396 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:43.355126 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.355531 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:43.360680 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:43.361564 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:43.370007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.373667 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.401332 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.491832 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.493327 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:43.493947 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.500985 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.504024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:43.513824 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.513840 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:43.598120 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:43.634033 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:43.634115 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.640573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:43.646631 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:43.651787 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.656348 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:43.658262 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240502 08:00:43.664052 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.672865 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:43.674356 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:43.675223 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.682031 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.702435 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:43.719333 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:43.721386 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:43.725019 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:43.764904 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:43.848011 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:00:43.950538 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:44.046579 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.048331 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.053509 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.058213 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.060892 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.075394 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.076668 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.090103 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.090392 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.099169 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.100427 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.107338 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.107884 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.149725 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.233691 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.236624 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.237946 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.247543 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.248095 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:44.252328 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:44.254000 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.258307 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:44.273623 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:00:44.287776 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:44.320569 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:44.364735 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.367586 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.367933 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.368322 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.388887 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.390197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.420907 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:44.431210 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:44.476655 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.477607 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.483848 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.490610 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.490969 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:44.496922 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:44.502115 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.514410 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.577448 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.578197 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.583019 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.589484 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.595901 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.603688 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.605706 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.714485 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.715176 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:44.718479 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:44.720330 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:44.726527 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:44.727449 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.739233 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.745743 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.762897 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:44.773303 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:44.776206 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:00:44.787245 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:44.867347 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.869084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.871527 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.888334 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.890326 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.899446 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.920950 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.937028 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:44.937292 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:44.937355 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.938434 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:44.939556 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:44.948993 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:44.964756 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.069435 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:45.071287 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.075053 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:45.077235 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 08:00:45.077762 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.085350 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.087963 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.090761 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:45.102800 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:00:45.202554 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:45.202605 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.206662 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:45.226972 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.228816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:45.231508 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:00:45.237825 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:45.243641 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:45.246435 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.310401 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:45.310753 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:45.312752 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:45.312820 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:45.321415 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:45.322444 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.323218 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.344796 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:45.347613 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:45.410818 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.412725 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:45.415246 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.417695 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:45.418227 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:45.423439 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.441390 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.538107 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.542410 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:45.546460 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:45.546537 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.550712 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.559676 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.560356 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:45.629932 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:45.669545 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:45.670801 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:45.674218 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:45.674573 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.681833 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.687490 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.688032 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:45.719022 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:45.735754 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:45.881837 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.883898 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:45.884263 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:45.897653 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:45.898818 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:45.908900 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:45.909004 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:45.910638 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:45.958703 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240502 08:00:45.958725 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240502 08:00:46.006323 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.011019 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.013909 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:46.017155 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.032285 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.036284 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.055464 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:46.073817 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:00:46.093050 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:46.097255 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.099555 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.108443 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.112241 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.122087 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.133749 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:46.187327 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:46.230091 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.238437 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:46.239861 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.242465 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.242930 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.252485 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.267058 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:46.289103 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:00:46.310570 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:00:46.376878 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.379951 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.381506 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.394459 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:46.402974 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.413420 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.420439 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:46.431392 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.433238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.434614 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.450047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:46.450371 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:46.453137 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.461710 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.464711 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:46.477094 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:46.575249 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:46.579798 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.584049 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.584616 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.590795 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.599004 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.621742 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.714931 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.716549 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.720146 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.727052 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.727833 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.751101 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:46.765148 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:46.772605 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.825006 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:46.829578 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:46.830775 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.833483 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.836098 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.841926 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.844257 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.856992 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:46.860852 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:00:46.907663 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:46.920318 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:46.922230 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:46.935171 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:46.936043 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:46.937637 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.938313 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:46.953176 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:46.965070 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.055039 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.059358 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.059747 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:47.066551 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:47.073781 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:47.074872 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.076617 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.095338 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.193734 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.194146 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.194458 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.204713 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.209359 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:47.232601 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:47.234032 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.245081 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:47.251850 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:47.271303 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.283306 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.288055 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:47.288522 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:47.290699 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:47.293010 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.303855 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.304387 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.411983 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.415675 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.417207 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:47.417452 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.436134 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.448916 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:47.449709 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:47.511059 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:47.558087 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.561435 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.567775 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:47.570477 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.570523 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:47.584066 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:47.588600 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:47.588892 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:47.676527 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:47.749775 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.752280 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.762161 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.762476 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:47.763622 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:47.763913 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.765805 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:47.831605 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:47.902499 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.903569 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.908963 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.909759 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:47.937080 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:47.937888 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:47.941447 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.960287 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:47.963807 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:47.966799 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.970518 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:47.984238 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:47.984751 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:47.997341 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:47.999228 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:48.093510 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:48.110574 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.111171 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.113637 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.113992 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:48.128448 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.128985 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.141196 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:48.209528 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:48.241755 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.245934 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.252740 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.255059 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.256018 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.256085 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.262326 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:48.264284 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240502 08:00:48.325305 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:48.373154 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.373154 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.374001 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.381814 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.383073 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.383137 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:48.408427 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.438428 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.444222 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.457991 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.461009 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:48.468315 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.481441 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.489360 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:48.576265 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240502 08:00:48.585583 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.593341 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.595752 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.596940 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:48.601142 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.614787 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.616025 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:48.670684 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:48.717804 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.720292 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.721442 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.726698 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.728708 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:48.731513 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:48.740020 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:48.741468 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:48.767467 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.810526 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.818382 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.819224 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.819779 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:48.820051 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.821127 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.823544 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:48.920413 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:48.924763 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.929545 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:48.931947 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.947772 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:48.952356 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:48.959165 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:48.968245 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:49.059518 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.062899 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:49.063189 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:49.063195 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.068235 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.079030 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:49.079864 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.196485 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:49.209770 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:49.215092 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.220677 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:49.231110 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.234177 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.237311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:49.238020 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:49.269994 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:49.271775 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:49.275352 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:49.276754 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.277909 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.286660 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:49.297397 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.307078 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:49.315798 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:49.370515 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:00:49.391956 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:00:49.491891 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:49.667886 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:49.670049 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:49.671052 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:49.679731 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.682051 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.688127 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:49.697592 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.703575 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:49.725804 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.735630 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.736492 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:49.739882 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:49.742147 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:49.747491 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.749606 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:49.750813 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:49.754040 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.883668 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.884874 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:49.886155 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:49.886965 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:49.891299 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:49.900730 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:49.905844 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:49.929505 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:49.930476 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240502 08:00:50.022887 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:50.033965 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.036603 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.039955 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.045217 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.048704 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.051517 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.072302 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.106845 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.113109 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:50.115227 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:00:50.116386 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.122303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.124630 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.134724 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.137682 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:50.237119 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:00:50.248873 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.249648 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.254684 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.257251 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.258255 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.260290 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.283524 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:50.305138 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 08:00:50.311445 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:50.342020 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:50.377434 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.379386 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.381968 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.384634 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.388712 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.390235 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.392808 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:50.495914 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:50.501436 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.501708 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.502938 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.504406 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.509238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:50.524433 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:50.529860 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.536592 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:50.542454 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:50.573690 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.583240 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.585587 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.589489 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.596411 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.614876 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.619103 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:50.697678 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:50.718531 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.725339 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.726308 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.731451 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.735227 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.756573 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.759828 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:50.809269 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:50.842187 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240502 08:00:50.871870 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.880156 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.881531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.890882 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.893412 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.900815 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.910656 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.911288 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.923827 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:50.926553 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:50.932883 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:50.943711 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:50.956805 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:50.957372 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.069154 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.077950 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.078234 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.079126 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:51.081799 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.099272 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.110303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:51.151458 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:51.209182 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.209916 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.215205 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.215669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.216909 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.230342 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:51.241971 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:51.276279 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:51.287590 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:51.320006 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.320993 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.324174 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.325142 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.332811 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.336445 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:51.357971 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:51.436316 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:00:51.475868 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:51.550004 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.552595 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.555543 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:51.555948 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:00:51.560519 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.579419 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:51.582890 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:51.591396 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.593008 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:51.674489 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:51.692351 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.695673 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.702296 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.704483 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:51.706691 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.709043 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.737932 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.782141 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.783668 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.787994 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.798784 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.799894 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:51.811666 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.817143 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:51.854849 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:00:51.859792 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:51.903494 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:51.917855 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:51.920532 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.920789 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:51.932719 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:51.933542 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:51.957334 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:51.957909 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:52.050217 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:52.072310 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.080354 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.081878 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.084055 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.084108 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.092725 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.097738 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.115420 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.117652 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.121910 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.129315 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.138783 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.150254 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.170943 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:52.179024 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:52.246166 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:52.263329 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.264544 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.264962 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.269359 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.272967 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.281044 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.283357 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.387530 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.388697 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.389961 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.390622 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.390877 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.399188 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.410871 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:52.481086 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:52.522861 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.522899 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:52.526523 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:52.533288 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.536913 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.548415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.558241 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.559439 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.579293 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.581373 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.590307 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.594635 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.600693 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.602046 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.626549 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:52.635265 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240502 08:00:52.675055 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:52.695540 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:52.720522 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.728282 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.731789 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.732308 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.735996 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.737681 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.744800 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:52.787428 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:52.865566 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:52.866755 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.872184 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.881465 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.897688 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.903364 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.907637 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:52.960637 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.965780 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:52.978407 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:52.980120 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:52.982439 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.001175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:53.007666 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:53.014050 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:53.080518 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:53.093454 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.100638 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:53.101289 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:53.102371 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.109217 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:53.113135 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:53.116192 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.128623 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:53.285224 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:53.329070 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.331728 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:53.333411 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.338485 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:53.341660 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.342572 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.372992 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.443487 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.447026 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.448279 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:53.461644 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:53.462152 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:53.462710 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.463225 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:53.473666 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240502 08:00:53.476213 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.574016 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.579737 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:53.581961 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.588425 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.588531 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:53.590706 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.625191 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:53.663707 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:53.717351 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.720558 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:53.721495 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.727001 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.737867 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.739398 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.749783 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:53.806414 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.808125 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.810626 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:53.811400 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.812901 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.836640 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.839488 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:53.878360 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:53.936810 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.938993 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:53.941223 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:53.944212 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.960140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:53.962435 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:53.975785 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:54.049356 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:54.065356 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.072322 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.078105 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.081877 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.103422 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.125638 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.127987 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.130520 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.132041 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.133987 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.142791 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.144001 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.147137 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.150604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:54.183912 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:54.205772 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:54.274291 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.278928 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.280659 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.281049 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.282400 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.293557 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.297399 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:54.321010 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:54.342828 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:54.358990 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:54.410759 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.412649 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.416813 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.431063 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.443611 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.446851 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.459888 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:54.467741 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:54.508944 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.511451 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.513404 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.514410 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.520676 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.532002 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:54.552595 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:54.558745 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.602291 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.603883 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.610145 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.613354 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.619625 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.623464 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.624167 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:54.720602 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:54.731778 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.732581 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.733402 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.735087 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.740799 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.747349 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:54.753780 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:54.777799 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.860863 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.861608 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.871415 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.874975 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.880105 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.889649 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.891435 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.943861 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:54.947268 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.949000 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:54.953194 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:54.956853 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:54.966461 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:54.976322 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:54.997161 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:55.086211 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:55.091959 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:55.096720 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:55.192281 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:55.214280 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.218075 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.218351 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.227804 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.228623 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.237403 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.250664 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:55.302421 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:00:55.303030 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:55.339701 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.342979 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.343166 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.344240 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.345409 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.346138 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.349135 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:55.354988 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240502 08:00:55.394492 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:00:55.396708 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 63 similar messages]
I20240502 08:00:55.465731 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.466420 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.473126 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.476002 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.479970 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.483140 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.485709 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.577886 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.581893 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.594909 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.600399 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.602999 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.611794 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.622138 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.670136 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.675428 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.678036 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.682453 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.682762 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.699273 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.713055 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:55.782754 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:55.809500 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.814029 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.814312 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.816917 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.830312 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.831705 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.847713 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:55.881058 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:55.939260 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:55.970165 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.973533 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:55.974238 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:55.975108 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:55.975340 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:55.977082 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:55.977785 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:55.984215 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:55.995393 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.001585 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.004653 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.010960 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:56.018158 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:56.020138 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240502 08:00:56.026475 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.044404 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.146824 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.153950 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.154078 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.154183 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:56.165823 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.167541 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.184268 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:56.233928 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:56.281528 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.281754 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.287197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:56.293553 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:56.298942 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.300136 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.303401 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.303933 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:56.321790 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:56.395882 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.396274 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:56.400708 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.405714 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.425568 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.429153 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.435771 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.478359 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.479307 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.485548 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.487150 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.490373 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:56.495890 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.515559 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:56.606985 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:56.611016 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.612547 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.613673 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.616816 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.618036 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:56.626999 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:56.629050 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.644646 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:56.714908 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:56.730202 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:00:56.731043 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:00:56.772790 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:56.793527 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.792412 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:56.799793 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.800352 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:56.801844 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:56.803701 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.821753 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.836409 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.841765 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.842451 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.843926 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:56.848870 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:56.859136 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:56.866463 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:56.914065 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:00:57.034323 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:00:57.110646 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:57.164042 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:57.165675 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.172526 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.189196 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.196429 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.198779 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.200523 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.237321 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:57.254802 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.257326 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.261438 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.266353 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.286219 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.295599 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.299644 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:57.385958 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.387341 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.388152 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:57.390218 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:00:57.392225 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:57.394937 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.400288 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.420606 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:57.453145 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:00:57.519454 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:57.520097 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.522044 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.525440 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.525907 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.525919 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.541438 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:57.541705 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:57.548985 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:57.582901 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:57.660810 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.672240 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:57.677457 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.677726 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.681164 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.689163 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.708778 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.720196 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.727769 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:57.728489 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:57.741624 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:57.742491 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.744612 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.764601 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.770268 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:57.801316 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:57.819550 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:57.866851 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:57.869403 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.870012 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:57.877799 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.879024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:57.888757 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:57.889847 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:57.998318 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.001169 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.006175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:58.018616 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.019134 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.019680 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:58.032567 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.104207 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.107743 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:58.109838 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.122568 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.127161 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.133764 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.135054 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:58.165711 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:58.201763 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.202191 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:58.203215 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.203438 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:58.203694 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.221958 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.232774 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.319905 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.322928 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.325718 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:58.326287 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.332314 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.332634 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.343238 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:58.363277 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240502 08:00:58.404413 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:58.446079 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:00:58.454082 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.455314 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.456568 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:58.468278 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:58.475919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.477891 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.488955 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.522222 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.526631 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.526997 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.527102 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.530212 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.530727 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:58.532856 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:58.626178 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240502 08:00:58.647728 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.652843 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.654702 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.655611 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.670064 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.677814 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:58.678563 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:58.717708 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:58.836108 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.838008 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:58.839762 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:58.842725 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.844995 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:58.855911 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.857195 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:58.946805 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:00:58.984028 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:00:58.993700 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:58.993731 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.001516 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.003535 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.004395 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:59.004508 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.004623 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:00:59.104250 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:00:59.135429 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.135581 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.140439 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.140923 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.151813 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.154662 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:59.181074 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:59.194129 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:59.228675 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.232442 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.235451 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.246098 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.251319 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.251787 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.266325 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:59.289202 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:00:59.329211 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:00:59.352774 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.354216 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.356112 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.357017 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:00:59.366617 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:59.367244 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.367573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.383914 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:59.434878 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:00:59.478062 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.479238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.481290 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:59.486198 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.488059 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.488322 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.519058 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:59.573513 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:59.586982 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.590642 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.591348 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.592813 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.600816 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.605749 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.634358 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:59.676122 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.678628 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.682109 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.682399 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.689671 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.704640 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.719303 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:00:59.793236 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:00:59.812250 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.812655 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.815944 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.817075 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:59.820365 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.837926 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.845839 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:00:59.948741 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:00:59.962805 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.968206 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.968741 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:00:59.972616 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.977306 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:00:59.978978 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:00:59.979483 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:00:59.982180 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.992512 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:00:59.996948 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:00.000270 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.011976 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:00.031414 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:01:00.034965 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:00.035037 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.036649 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:00.050755 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240502 08:01:00.133724 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:01:00.137676 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:00.138651 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.140565 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.149098 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.150067 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:00.166396 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.170943 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:00.258222 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:01:00.270002 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:00.270076 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.276626 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.290620 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.291105 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.315490 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.317426 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:00.338582 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:00.339622 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:01:00.411108 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.417728 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:00.430224 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.431001 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:00.433229 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.445657 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:00.447160 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:01:00.454689 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:00.464581 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:00.470013 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.497493 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.501590 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:00.505750 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.520182 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.521142 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:00.523461 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.525561 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:00.677017 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:00.775521 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.779496 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.782850 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:00.791013 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.801563 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:00.801918 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:00.817616 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.824988 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:00.852844 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:01:00.879524 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:00.881273 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:00.881762 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:00.884076 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.886933 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.887902 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:00.888654 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:00.914012 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.005678 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.008229 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.013273 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.020630 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.021332 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.029089 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.029525 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:01.066290 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:01.081571 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240502 08:01:01.140518 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.146481 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.149009 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.157773 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.159691 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.160537 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.168085 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.178557 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.182855 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.190598 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.191982 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.196825 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.221956 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.226152 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.330229 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.331867 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.332216 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.337942 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.352069 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.354693 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.365694 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.458952 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.461216 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.461270 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.464299 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:01.465087 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:01.475893 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.488840 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.490327 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:01.522639 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:01.579298 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.582084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.586144 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.600378 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.603976 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.607616 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.621891 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.650003 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.652514 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.662171 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.663391 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.672663 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.673580 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.676081 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:01.740983 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:01:01.785184 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:01.789831 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.793455 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:01.796689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:01.801481 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:01.804955 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.811520 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.816730 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.828753 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:01.909365 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:01:01.910128 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:01.918850 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:01.923051 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.927134 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:01.930666 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:01.938824 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:01.950881 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:01.952973 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:01.964844 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:01.989250 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:02.021282 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.026357 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.032886 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.035065 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.036015 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.040611 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.060113 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.124336 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.127171 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.127724 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.128121 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:02.131317 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:02.135243 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.135658 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.162382 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:02.170203 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:01:02.254530 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.255760 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:02.256440 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:02.256737 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.260881 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.271741 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.276649 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.285776 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.380342 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.380420 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.382445 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.383361 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.388442 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.392657 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.409384 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:02.441471 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:02.578931 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:02.604075 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.610519 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.611377 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.614687 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.615346 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.631808 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.650856 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:02.678153 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:02.733851 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.734908 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.748970 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.752089 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.754272 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.762377 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.784749 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:02.801316 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:01:02.815691 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:02.816416 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.823415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.825194 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.826126 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.827663 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.831928 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.836642 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:02.841176 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240502 08:01:02.868659 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:02.948792 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.953083 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:02.953864 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:02.954953 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:02.959928 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:02.960448 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:02.985754 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:02.999598 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:03.088160 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.088698 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.096143 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.096917 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.110152 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.110586 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.139545 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.142849 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.149123 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.155761 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.160676 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:03.169756 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:03.175583 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.179490 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:03.183185 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:03.183884 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.303315 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.305044 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.306862 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.316417 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.327316 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.332938 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:03.343729 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:03.345117 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:03.386672 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:03.436367 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.436497 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.441301 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.446249 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.454756 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.456630 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.472044 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.548367 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.554682 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.555775 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:03.557641 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240502 08:01:03.560269 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.561811 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.565054 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.568034 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.641196 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.643610 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.644073 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.644070 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.647648 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.648660 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.679860 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:03.689920 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:03.767838 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.773277 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.780198 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.783944 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.787887 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.789605 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.805814 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:03.837584 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:03.902516 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.908907 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.909821 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.921305 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.921600 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.926436 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.938045 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:03.939108 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:03.946027 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.950363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.950780 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:03.960147 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:03.961306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:03.989976 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:04.060973 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:04.091598 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.094583 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:04.095175 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:04.101538 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:04.104887 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.111245 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.113338 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.114461 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:04.117581 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:01:04.259063 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:04.290540 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:04.292240 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.299085 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:04.301088 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:04.304647 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:04.305833 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.319481 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.325958 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:04.373584 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:04.432369 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.432929 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.433296 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:04.433925 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:04.442823 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.455063 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:04.460825 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:04.507323 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:04.568122 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.576913 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:04.578902 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.588172 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.589205 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:04.599403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:04.623346 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.679183 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.681308 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.689206 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:04.691565 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:04.698081 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:04.698766 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:04.699472 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:04.709120 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:04.728523 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:04.739658 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:04.777426 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:04.777866 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.783797 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:04.790267 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.800999 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.805492 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:04.811722 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.904876 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:04.906544 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.910781 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:04.913374 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:04.914994 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:04.925431 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:04.937366 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:05.029656 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:05.052961 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.059993 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.061674 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.062055 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.066105 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.068871 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:05.075845 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.089772 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.092953 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.109803 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.110190 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.127403 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.131726 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.137041 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:05.195971 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:05.240361 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.243904 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.249657 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.249979 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.251858 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.264123 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:05.264621 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:05.265360 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:05.269722 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:05.276957 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:05.289741 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:01:05.368984 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:05.371785 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240502 08:01:05.376196 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.383378 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.384023 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.386551 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:05.388269 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.390036 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.423487 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.482038 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.486819 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.487311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.495379 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:05.503216 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.505317 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.517887 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.553932 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.558437 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.560173 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.566538 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.572948 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:05.595618 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:01:05.599402 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.599795 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:05.669942 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:05.695379 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.699015 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.701113 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.701318 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.708843 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:05.714597 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.725615 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:05.748206 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:05.748448 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:05.835573 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.836941 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.838812 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:05.839560 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.844628 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.852726 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.858681 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:05.871102 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:01:05.877264 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:05.908054 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:05.911758 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.915453 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:05.916633 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:05.932765 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:05.934047 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:05.949374 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:06.160914 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:06.173684 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240502 08:01:06.186399 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:06.187741 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.192749 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.194093 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.201359 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.208086 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.212779 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:06.228780 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.313423 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.318035 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.319983 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.322876 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.332249 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:06.339619 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.367992 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.394752 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.394806 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.397145 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.398121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.415095 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:06.423437 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:06.425689 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:06.426010 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.529712 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.530586 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.531474 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.541296 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.549513 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.560532 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:06.560815 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:06.606899 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:06.672689 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.673159 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.675413 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.676718 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.687498 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.700323 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:06.706081 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:06.732129 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:06.736614 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.738924 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.753718 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:06.755226 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:06.758038 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:06.765687 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:06.769809 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:06.769960 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:06.770213 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.781297 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.885707 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:06.887612 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.892576 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.893949 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:06.900866 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:06.904697 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:06.911824 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.015507 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.015965 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.018599 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.036957 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.038858 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.051985 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.058996 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:07.076083 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:07.130681 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.130993 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.131503 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.135864 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.140476 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.145653 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.151569 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:07.154560 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:01:07.180584 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:01:07.190622 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:07.217974 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:07.235248 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.235597 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.235873 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.242411 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.248659 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.251688 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.265434 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.350942 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.351591 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.351814 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.356935 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.368153 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:07.379441 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:07.380257 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.385612 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:07.416568 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:07.479306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.485128 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.490038 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.499266 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.504056 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.505669 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:07.534467 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:07.540166 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.549875 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.552376 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.552671 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.553380 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.554822 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.567385 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.595413 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:07.659210 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:01:07.673480 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:07.701870 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.702153 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.704231 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.705282 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.706102 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.708966 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.709756 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:07.737916 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:01:07.782426 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:07.805377 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:07.848198 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:07.851191 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:07.851851 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:07.860091 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:07.867161 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.877427 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:07.886162 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.027779 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.028967 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.029320 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.030748 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.033435 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:08.044478 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.045220 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.162998 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.166170 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.171514 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.172358 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.172760 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.181521 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:08.192483 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:08.197939 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:08.251914 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.253335 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.259688 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.259688 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:08.271601 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.272775 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.278151 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.363242 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.364926 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.368026 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.369755 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:08.373595 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:08.380299 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.390909 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.399971 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:08.409705 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:01:08.444403 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240502 08:01:08.501473 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.503093 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.503929 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.504041 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.510437 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:08.517081 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.521397 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.628661 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.637144 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.640702 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.643285 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:08.652611 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.670929 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.675542 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.684441 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.688753 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.696707 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.699862 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.712939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.728566 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.739249 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:08.746577 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:08.788520 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240502 08:01:08.789059 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:08.849133 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.851068 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.851688 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:08.853128 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.854727 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.856431 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:08.868809 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:08.897495 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:08.980355 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:08.983569 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:08.986604 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:08.987326 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.002820 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.013502 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.015269 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.076884 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.079114 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.079504 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.093003 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.093672 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:09.105769 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.107514 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:09.139832 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:09.168233 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.169198 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.179211 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.187157 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:09.187984 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.195187 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.205480 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:09.233951 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:01:09.259199 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:09.299651 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:09.316166 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.318238 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.318714 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.320609 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.324330 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.329609 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.332369 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:09.358798 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:01:09.372687 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:09.459578 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.461282 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.462999 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.476505 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.483247 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.496676 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.498932 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.503942 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.506758 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:09.508294 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.517560 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.521775 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:09.526511 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.548852 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:09.651460 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:01:09.716655 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:09.862843 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.865439 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:09.869534 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:09.875857 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.890005 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:09.895681 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:09.897441 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.903268 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.907101 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:09.921414 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.933743 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:09.938967 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.938967 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:09.950212 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:09.973806 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.015349 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:10.058794 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.060881 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:10.061182 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.061249 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.067974 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.071555 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.085415 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.181378 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:10.184993 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.190989 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.194380 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.194532 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.195770 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.210183 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:10.211483 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.217980 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:10.261380 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:10.264838 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:10.330926 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.334712 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.338059 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:10.338991 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.346743 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.354262 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.368492 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:10.372781 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.392691 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:10.395395 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.417155 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.433149 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.433197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.472438 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.472679 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.477175 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:10.567626 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.571185 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.574139 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.575584 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.580297 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.592273 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.593636 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:01:10.605609 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:10.662024 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:01:10.711124 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:10.715204 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.720677 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.722101 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.722716 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.726150 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.726521 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.771190 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:01:10.816620 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.818322 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.819011 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.823158 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:10.830637 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:10.831854 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:10.832147 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.832686 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:10.845649 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:10.903967 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:10.910291 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:10.910925 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:10.910952 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:10.912587 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.925910 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:10.934788 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:10.976269 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:11.033597 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.034828 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.037312 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.039149 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:11.039743 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240502 08:01:11.041503 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.067333 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.069890 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.156486 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.165689 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.167352 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.169289 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.170163 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.178191 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.184046 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.246155 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.248221 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.249336 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.250203 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.251495 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.264118 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.271494 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:11.331384 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240502 08:01:11.334705 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:11.387249 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.389572 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.389691 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.389842 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.394481 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.404533 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.407614 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:11.436800 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:01:11.479808 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:11.511348 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.511605 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.515122 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.518316 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.528388 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.541563 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.549743 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:11.550073 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:11.706215 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:01:11.709026 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:11.727006 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.730930 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.731243 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.732105 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.738152 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.746306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.782008 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:11.821796 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:11.862865 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.863423 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.863852 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.863974 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.869004 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.868988 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.897375 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:11.901260 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:11.946987 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:11.949180 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:11.950306 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.952858 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:11.967914 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:11.979733 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:11.982877 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.074594 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.075847 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.079706 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.091868 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:12.092396 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:12.093056 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.093690 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.113960 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.230417 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.230727 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.233601 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.233892 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.234544 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.241243 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.246594 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.270375 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.270825 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.272941 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.295450 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.316843 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.317018 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.318892 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:12.405110 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:12.426463 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.429903 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:12.436793 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:01:12.442799 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.444087 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.444353 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.445998 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.447667 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:12.498481 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:01:12.514103 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:12.554910 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:12.560696 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:12.561277 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.562235 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.564304 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.569065 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.574434 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.578547 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.584744 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:12.625756 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240502 08:01:12.657869 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:12.669572 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.672953 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.683027 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.684625 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.693161 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.707996 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.711458 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:12.721675 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:12.772838 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.778465 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.781070 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.792310 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.793190 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.795184 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.813060 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.900539 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:12.901461 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.902321 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:12.905872 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:12.915169 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:12.918535 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:12.919014 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:12.989362 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:13.029639 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.038589 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.043032 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.043190 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.045221 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.064085 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.068324 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:13.085794 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:13.104235 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.105741 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.106606 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.109911 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.126329 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:13.130575 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.131747 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:13.132270 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:13.247670 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.252038 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.254235 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.254235 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.260630 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:13.265317 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.267081 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:13.320099 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:13.392833 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.397670 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:13.399179 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.400624 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.403558 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.413340 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.433297 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:13.479295 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:13.567919 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:13.583120 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.588279 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.599447 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:13.599927 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.602183 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.609652 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.626084 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:13.648447 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:13.741680 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.743129 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.743675 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.757216 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.757721 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:13.760190 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:13.767977 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.789836 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.796527 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.807060 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:13.809327 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.822032 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:13.830703 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:13.831499 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.840195 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.943676 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:13.948859 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.952811 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:13.954391 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.956142 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:13.957932 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240502 08:01:13.962720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:13.979175 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:14.025382 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:14.059351 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:14.073217 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.073475 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.077280 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.081753 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.086215 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:14.087345 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.088162 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:14.091338 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:14.200661 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.206756 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.207605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:14.214128 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:14.216310 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.223728 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:14.251205 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.256901 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.273581 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.283454 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:14.286072 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:14.296995 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.297349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.301468 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.309211 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.310849 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:14.420131 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.425451 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.428630 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.431250 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.432603 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.444195 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.445834 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:14.452564 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:01:14.536821 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:14.552277 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.555223 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.556427 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.558701 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.561798 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:14.568856 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.571156 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:14.596920 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:14.670051 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.674587 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.676640 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.677372 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.708446 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.709985 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.710383 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:14.754849 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.764379 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.770109 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.770215 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.771454 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.784049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.793614 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:14.875917 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:14.888001 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:14.889472 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:14.895500 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:14.900710 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:14.901387 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:14.922727 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:14.922818 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:14.935030 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:14.963164 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:01:15.000254 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:01:15.002334 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:15.042474 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.048907 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.052654 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.068259 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.068768 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.072662 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.085892 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.087837 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:15.091537 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.101879 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:15.117343 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.117398 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.129451 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.129577 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:15.256424 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:01:15.299706 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:01:15.367709 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:15.378561 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:15.454738 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.469930 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.470400 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.471581 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.478050 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:15.483141 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.483748 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.499369 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.503970 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.508805 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.526757 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.530155 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.533937 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:15.539180 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:15.541118 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:15.632726 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:15.647043 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.647578 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.647879 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.648805 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.649678 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.655377 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.660010 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:15.664644 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:01:15.713979 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:15.757119 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:15.781066 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.781384 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.784644 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:15.791374 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:01:15.796550 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.797336 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.800362 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.809458 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:15.897531 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.900884 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.908221 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:15.908502 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.909052 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.923447 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.941696 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:15.982995 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:15.983317 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:15.983585 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:15.988274 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:15.992678 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:15.994782 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.005489 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.007263 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:16.034214 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:16.096853 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240502 08:01:16.107937 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.114785 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.115341 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.116540 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.117130 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.120788 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.133198 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:16.171754 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:16.249267 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.251902 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.252714 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.256659 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.258285 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.261212 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.268616 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.335098 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.337580 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.341456 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.341786 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.343433 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.343659 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.380299 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.447857 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.448478 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.454567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.454653 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.471814 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.475880 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.478396 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.566828 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.570892 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.572607 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.580646 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.590898 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.591365 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.601033 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:16.622082 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:01:16.650518 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:16.661690 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240502 08:01:16.688092 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:16.719519 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.723572 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.725296 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.735209 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:16.745714 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:16.746631 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.750133 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.759837 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.764000 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.768865 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.774195 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.776084 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.781875 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.826881 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.827379 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:16.850667 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:01:16.856439 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:16.922503 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:16.923805 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.924873 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:16.927946 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:16.937300 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:16.954862 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:16.955688 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:17.068563 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:17.118885 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.119138 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.119913 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.120716 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:17.121208 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.121841 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.141965 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:17.181576 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:17.269187 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:17.269649 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:17.269790 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.273198 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.273977 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.289618 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.293519 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:17.297828 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:17.298169 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.399097 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.403355 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.408262 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.415449 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.417783 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:17.418090 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.433310 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:17.469276 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:01:17.480235 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:17.503937 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.506484 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.508767 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.513188 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.513456 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:17.521967 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:01:17.544155 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.550282 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.605854 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.605994 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.607316 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.608919 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:17.612655 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.620751 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:17.628574 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:01:17.630812 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:17.686008 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:17.731477 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.734320 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.740531 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.745592 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:17.749413 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.754613 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.755615 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:17.819307 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:01:17.837951 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:17.867358 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.869244 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.877897 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:17.879976 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.887653 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.888645 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.893862 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.934762 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:17.939604 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.939603 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:17.941308 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:17.951848 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:17.953742 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:17.957981 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:18.060859 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:18.076304 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.080514 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.081260 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.082832 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.090584 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:18.092823 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:18.113458 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:18.129616 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:18.211417 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.216694 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.219314 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:18.219779 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:18.223209 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.230702 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:18.233067 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:18.247593 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.347352 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.355754 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.357930 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.360193 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.365685 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:18.382400 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.385316 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.386689 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.396354 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:18.399706 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:18.422600 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.423478 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:18.425974 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.427296 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:18.435925 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.551276 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.551338 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:18.554672 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.556771 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.559363 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.563110 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:18.573012 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:18.599773 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240502 08:01:18.609089 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:01:18.675906 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:18.676882 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.680115 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.680967 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.689422 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:18.693333 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.709455 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:18.712733 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.786335 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:18.789530 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.791402 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.804831 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:18.813249 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:18.817207 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:18.836856 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:18.876111 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:19.017968 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.018323 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.020370 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.021785 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.022737 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.037132 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.041476 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:19.089982 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:01:19.124421 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:19.144358 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.144523 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.159461 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.169492 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.175489 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.189656 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:19.190111 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:19.194236 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.206178 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.225394 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.226719 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.228221 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.233520 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.235463 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.241710 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:19.252274 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:19.363504 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.364006 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.368031 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.374704 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.380656 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.388672 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.389250 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:19.437058 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:19.464987 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:19.494946 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.499486 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.505542 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.506392 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.514595 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.534126 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:19.543946 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240502 08:01:19.544782 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.598583 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.601776 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.608582 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.609048 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.617489 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.634243 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.637706 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:19.646483 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:19.713423 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.713970 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.715900 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.722092 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.726814 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.740077 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:19.744290 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:19.757158 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:19.825707 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:19.853873 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.854545 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.861124 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.862092 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.868286 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:19.879935 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:19.883240 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:19.887251 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:19.896962 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:19.973202 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.974365 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:19.975579 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:19.975925 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:19.992415 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:19.994794 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.000681 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:20.044400 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:20.067162 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.068910 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.070895 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.071893 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:20.079128 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.085253 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.088686 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.187551 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.188779 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.190004 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:20.194692 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.195806 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.196508 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.200651 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:20.213873 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:01:20.274916 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240502 08:01:20.275158 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:20.333832 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.335258 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.339484 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.344246 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.353801 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.356796 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:20.366863 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.377483 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.381983 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:20.392907 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:20.395709 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.403383 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.418802 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.425206 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:20.438023 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.550596 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:20.554077 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.561547 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.562193 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.563138 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.571224 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.583412 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.724097 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.724663 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.724943 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.725309 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.726365 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.746014 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:20.759480 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:20.782116 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:01:20.782895 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:01:20.792634 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:20.851226 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:01:20.864279 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:01:20.888764 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.890635 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:20.891330 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:20.897912 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.898998 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:20.903545 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:20.934440 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:20.955996 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:21.031569 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.035977 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.045969 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.046219 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.050540 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.052919 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.068832 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.129359 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.133790 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.137225 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.141794 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.150235 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.176003 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.179760 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:21.188572 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:21.236375 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.238191 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.239034 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.239070 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.242609 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.254197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:21.260588 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:21.275640 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:21.329895 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:21.366461 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.367559 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.377831 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.377945 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.381062 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.383354 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.404642 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:21.410300 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:21.486611 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.487669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.489135 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.496560 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.508297 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.510362 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.510828 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.571768 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.576962 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.583227 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.586452 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.589571 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:21.590701 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:21.599560 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.615147 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:21.646726 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:21.710321 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.710417 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.716674 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.717769 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.720356 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.730252 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.732108 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:21.740301 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:01:21.795154 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:21.846824 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.853096 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.853546 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.866462 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:21.867046 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.877990 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.879504 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.915269 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.915333 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:21.915840 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:21.917642 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:21.922454 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:21.922792 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:21.935581 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:21.944398 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:21.964825 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240502 08:01:22.045272 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.046581 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.048473 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.049083 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:22.053190 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:22.055487 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.064663 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:22.156615 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:22.184154 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:22.184967 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.185108 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.187752 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.201750 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:22.202409 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.208914 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:22.229532 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:01:22.271003 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:22.310956 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:22.310985 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:22.317976 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:22.318622 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:22.332569 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:22.338582 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:22.348011 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:22.358067 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.359699 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.363633 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:22.370721 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.371393 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.373540 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.376046 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.391556 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.407507 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:22.415511 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:22.525993 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:22.733883 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.735163 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:22.745738 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:22.750200 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:22.752437 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.757452 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:22.767303 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:22.770958 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:22.783385 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.787621 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:22.788558 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:22.791301 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:01:22.791448 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.797348 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.802354 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.803337 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.833644 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:22.881887 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:01:22.882493 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:22.929067 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:22.929054 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:22.931905 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:22.932591 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:22.940759 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:22.941553 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.967948 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:22.972260 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.066593 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.067919 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.074038 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.074913 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.079201 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.103291 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.109205 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.176729 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.180686 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.189756 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.191349 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.195802 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.196955 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.197485 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.261557 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.267315 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.269048 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.272226 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.277925 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.303963 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.306020 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:23.363210 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:23.400656 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.400749 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.409780 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.410982 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.412097 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.416666 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.432612 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:23.457870 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:01:23.504114 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:23.550767 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.551770 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.552469 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.555244 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.562323 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.570011 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.573173 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.573736 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.586241 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.599151 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.614164 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.614495 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.620570 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.648949 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.745590 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.746484 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.752887 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.758086 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.761390 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.771812 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.792433 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.880630 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:23.881280 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.886893 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.898430 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:23.900336 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240502 08:01:23.901366 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:23.908821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:23.922369 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:23.925125 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:23.944646 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:01:23.959002 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:23.980347 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:23.980729 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:23.990123 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:23.991492 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.012220 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.013926 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.030141 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:24.066015 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:24.077672 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.080039 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.083793 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.087491 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.088855 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:24.098546 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.098927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.214112 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.215406 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.222205 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:24.226697 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.231369 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.249307 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.252563 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:24.254226 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:24.339455 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.340621 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.343914 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.343910 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:24.358556 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:24.361186 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.373742 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.376961 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:24.495105 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:24.543207 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.551530 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:24.560959 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.561846 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.569239 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.569749 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:24.570158 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:24.571146 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.688374 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.694216 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.711608 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:24.714222 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:24.715310 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:24.717027 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.719175 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.728304 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.740479 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.744973 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.769693 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.777863 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.779058 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:24.784027 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:24.784376 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:24.791376 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:24.805567 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:01:24.813457 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240502 08:01:24.856858 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:24.906919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.907505 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:24.907774 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:24.908164 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:24.920403 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:24.923650 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:24.925285 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:24.928887 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:01:25.033005 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:25.037000 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.038430 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.044744 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.049082 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.056144 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.058475 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.059561 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.137311 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.139555 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.150521 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.155616 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.163877 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.189944 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.196563 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:25.206434 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240502 08:01:25.207715 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:25.238484 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.242527 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.247511 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.255714 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.257236 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.269264 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.284443 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:25.377733 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:25.386837 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.387549 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.390066 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.390908 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.391292 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.398867 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.403721 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:25.479310 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:25.527422 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.528221 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.538981 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:25.540546 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:25.546010 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.546272 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.553159 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.571941 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.574767 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.581236 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.589141 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.602164 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.615191 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.617076 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.618295 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:25.658617 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:25.719942 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.723379 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.725111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.725936 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.726531 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.736119 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.762938 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.857916 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.863745 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:25.865742 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 08:01:25.866235 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.868126 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.868147 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.869279 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.881855 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:25.965936 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:25.966637 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:25.973856 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.974553 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:25.988782 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:25.989876 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:25.995185 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:01:26.000593 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:26.031697 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:26.032523 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:26.060906 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.063024 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.065912 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:26.071319 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.074434 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:26.093344 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.095202 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:26.107656 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:01:26.190115 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:26.304621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.306658 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.308571 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.309800 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.320632 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.323127 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:26.328425 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:26.425060 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.427812 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.428364 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.428479 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:26.435096 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.459919 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.462011 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:26.501329 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:26.538589 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:26.556164 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.560771 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.564591 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.565644 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.565913 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.565990 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:26.566311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:26.587543 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:26.646776 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:26.689541 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.691151 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.699075 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:26.700856 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:26.724083 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.734012 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.739109 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:26.760448 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:26.766813 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.767282 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:26.776170 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.793532 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:26.808887 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.810879 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.811556 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:26.871377 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:26.903131 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:26.907631 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:26.909416 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.913856 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:26.927642 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:26.929826 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:26.932655 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:27.012918 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:27.043584 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.051431 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.056608 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.057346 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.057344 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:27.059569 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:27.062589 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.065591 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:27.139236 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:27.145092 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.145768 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:27.151731 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:27.153066 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.155846 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.156147 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.156148 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.180660 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:27.236891 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:27.254298 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.255362 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.260363 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.260666 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.264518 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.274181 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.304423 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:27.329368 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240502 08:01:27.394038 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.395303 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:27.396252 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:27.398558 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.400079 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.400851 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:27.411661 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:27.415356 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.427026 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:27.440963 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:27.531060 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.531751 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.541545 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.549942 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.558740 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.559011 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.559150 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.568888 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.573393 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.573804 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:27.589905 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:27.599335 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.600914 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.603682 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:27.630506 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:27.633020 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.736428 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.746748 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.747723 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.749784 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.750586 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.752092 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.752614 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:27.820729 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:27.845108 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:01:27.891127 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.892338 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.897507 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.899158 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.904659 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.911861 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.941473 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:27.985575 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.986279 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:27.987350 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:27.992823 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:27.993113 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:27.996517 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.016669 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:28.029696 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240502 08:01:28.043685 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:28.056361 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:01:28.057163 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:28.246734 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:28.249473 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.253508 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:28.262157 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.262706 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:28.275291 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.275769 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.390110 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:28.392359 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.400200 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:28.403465 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:28.404081 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.412657 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:28.413723 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.413975 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:28.418395 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.449894 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:28.450428 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.466472 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.485949 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.487028 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:28.487689 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:28.512732 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:28.628243 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:28.633302 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:28.635356 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.635692 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.643054 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:28.656216 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.703519 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:28.816927 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:28.828173 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:28.833004 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:28.838689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.840296 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:28.841022 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:28.847323 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:28.847972 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.862144 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.882467 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.885650 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.898151 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:28.900410 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:28.900741 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:28.904263 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:28.915570 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:28.933008 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:28.939567 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:29.043584 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.043632 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.044284 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.049880 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.056437 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.062443 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.071290 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:29.163434 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.165071 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.176544 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.178032 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.188962 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:29.195536 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:29.198663 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:29.215423 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:29.218233 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:29.229658 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:29.289093 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.293258 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.293314 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.299124 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:29.306865 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.307919 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.317864 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:29.372268 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:29.378832 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.384645 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.385954 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.387571 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:29.394064 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.402022 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:29.419350 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:29.420351 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.504693 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.507016 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.507429 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.508709 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.517943 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.533511 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:29.533690 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:29.557463 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:01:29.649055 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:29.649811 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.655299 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.659806 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.666421 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.671628 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:29.674575 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.691450 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.693351 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.695676 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:29.697016 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:29.702242 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.702987 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:29.705487 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:29.707638 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.715333 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.746198 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:29.786260 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240502 08:01:29.851689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.852090 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:29.863075 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:29.864771 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:29.867355 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:29.868942 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:29.889088 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:29.918952 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:29.928723 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240502 08:01:30.038182 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:30.040518 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.043481 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:30.043905 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.051021 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.052019 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.060176 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.061925 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:30.168530 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:30.189821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.191658 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:30.193769 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.194170 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.194665 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.196962 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.222579 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:30.286221 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:30.324714 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.332521 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.340448 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.343326 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:30.355504 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.361639 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.376732 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:30.412947 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:30.422422 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:30.422878 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:30.426373 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.448979 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.454088 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.454538 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.455834 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.460980 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.527267 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:30.530860 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:30.535665 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.535984 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.537953 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.542564 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.543923 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.551090 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.677974 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:30.686841 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.686879 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.690971 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.699565 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.699642 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.715077 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:30.791446 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:30.815969 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:30.818846 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.829130 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.840298 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.848651 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.851590 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:30.852609 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:30.864635 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.870908 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:30.888857 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:01:30.899708 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.903692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:30.904731 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:30.904843 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:30.909381 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.017026 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.019795 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.022064 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.022830 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.040352 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:31.040859 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.041916 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:31.080905 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:31.125662 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:31.144412 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.144874 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.145869 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.147280 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.160480 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:31.161074 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.171202 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:31.172595 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:31.261863 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.263957 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.264416 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:31.266513 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:01:31.268007 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:31.270440 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.274533 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.274714 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.287441 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.345825 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.347841 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.362061 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.362078 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:31.362908 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.364341 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.374109 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:31.451685 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:31.482028 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.484588 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.485471 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.485898 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:31.501128 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.502765 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.503391 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:31.561962 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:31.615643 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.624856 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.624994 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.626368 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.627386 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.628098 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.633879 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:31.690153 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:31.706593 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.707940 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.718398 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.721421 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:31.721917 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.721915 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.740883 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:31.840482 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:31.885540 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:31.956147 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.958338 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:31.961678 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:31.964654 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:31.966502 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:31.968377 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:31.987695 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:31.997277 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:32.060245 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:32.076376 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.077435 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.081602 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.092876 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.106345 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:32.110357 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.127557 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.162209 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.162492 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:32.168409 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:32.172734 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.180631 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.189930 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.193907 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:32.195173 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:32.222048 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:32.272615 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240502 08:01:32.300979 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.304100 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:32.305517 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:32.308497 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.318658 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.319506 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.319902 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.335240 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:32.447721 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:32.452236 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.456629 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.467638 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:32.472905 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.480619 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.484417 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.486284 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.486477 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.490828 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.493342 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.521384 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.528664 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.537678 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.540565 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:32.551609 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:01:32.578198 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:01:32.624078 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:32.642056 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:32.652933 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.658296 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.659449 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.661355 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.663095 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:32.670877 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.678576 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.805070 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.805594 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.817423 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.823444 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:32.826426 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.835637 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.836560 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.903999 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.910404 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:32.917825 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.917966 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:32.919829 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:32.936856 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:32.956022 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:32.996613 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.002648 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:33.009090 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:33.014259 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.015388 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.022043 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:33.025305 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:33.026679 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:33.038081 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:33.064409 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:01:33.111088 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:33.125449 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.129278 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.129614 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.132529 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.138541 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.141983 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.147508 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:33.155366 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:01:33.280635 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.281457 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.285774 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.296415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.300045 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.306852 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:33.312803 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.312978 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.324208 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.332518 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.334856 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:33.337955 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.340840 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.353360 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:33.432368 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:33.472646 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.472924 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.473801 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.479879 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.484076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:33.499307 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:33.507560 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.510151 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:33.525415 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:01:33.595317 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:33.669358 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.674088 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.674850 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.679016 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:33.680555 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.686837 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.700475 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.802979 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.803308 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.813109 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.824630 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:33.824967 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.825729 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.826498 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:33.859941 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:33.932407 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.940833 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:33.941707 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.942934 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:33.948061 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:33.948601 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:33.948849 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:33.974272 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:34.030138 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:01:34.052130 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240502 08:01:34.053277 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.060525 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:34.060802 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.061357 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.069278 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.085878 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.090920 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.141561 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.142557 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.146091 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.146809 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:34.148237 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.158684 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.171061 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:34.253870 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:34.269402 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.274969 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.275365 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.277225 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.283725 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.288753 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.297567 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:34.392700 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:34.400288 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.403496 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.404256 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.405632 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.410071 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.417528 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:34.432595 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:01:34.433588 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:34.478817 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.479324 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.489547 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.491927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.493976 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.493986 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:34.512677 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:34.514597 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:34.535667 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:01:34.596769 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:34.615715 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.617172 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.618873 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.619433 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.620146 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:34.621438 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.639669 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.756755 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.759606 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.764261 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:34.765885 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:34.779876 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.789952 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.808640 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.811313 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.816848 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:34.819483 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240502 08:01:34.826941 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:34.828596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.829576 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.853610 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.859846 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:34.872653 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.875949 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.945389 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.946702 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:34.952412 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:34.954089 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:34.959437 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.961738 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:34.964334 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:34.979846 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:35.005301 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:35.078368 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.082448 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.088693 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.093084 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.093572 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.101716 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.105454 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:35.121214 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:35.221722 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.237270 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.238643 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.247913 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.256801 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.257302 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.261545 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.263113 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.269131 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.271631 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.273778 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.284902 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.307689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:35.326411 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:35.335157 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:35.395053 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240502 08:01:35.419355 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:35.600718 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:35.642134 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.644860 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.646389 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.650319 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.662223 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.664647 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.680528 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.704859 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.714280 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.714936 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.716164 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.721021 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.725836 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.736995 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:35.844522 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:35.844780 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.854532 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:35.854816 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.857415 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.859319 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.863495 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.864876 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:35.916786 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:35.939349 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:01:35.978339 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:35.983951 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.987102 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:35.991825 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:35.995363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:35.997018 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.008855 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:36.051733 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:36.087231 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.088790 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.090178 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.096690 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.108651 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.124749 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.126379 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.163523 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.166587 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.173034 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.173282 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.179665 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.183028 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.184623 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:36.219218 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:36.276934 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:01:36.300426 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.304379 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.311084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.312049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.322635 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.327746 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.327874 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:36.337404 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:36.354698 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:36.450779 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.457273 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.458002 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.467754 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.469985 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.479213 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.488689 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.491604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.510134 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.513346 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.513978 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.520237 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:36.521844 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:01:36.523840 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:36.524304 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.526360 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:36.628392 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:36.644661 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.646481 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.646737 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.649816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.652112 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.677028 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.681298 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:36.728130 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:36.771061 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.778558 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.780536 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.781522 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.793526 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.801880 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.826839 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.908066 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.911197 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.914777 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:36.917217 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:36.928818 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.929149 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.944417 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:36.962175 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:36.988055 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:36.994778 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.997980 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:36.998673 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:36.999315 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:37.007653 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.007975 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:37.028610 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:37.113286 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.114866 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.117419 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.119403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:37.120409 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.137372 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.143707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:37.217777 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:01:37.243801 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:37.265260 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.266070 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.276659 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:37.277729 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:37.278358 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.290092 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:37.291093 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:37.300822 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.316393 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:37.341823 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:01:37.429988 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240502 08:01:37.444676 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:37.449529 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.453258 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.454851 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:37.460274 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.460554 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.460851 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:37.463762 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:37.476727 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240502 08:01:37.579911 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.581888 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.583828 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.590726 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.599707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:37.601714 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:37.614132 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.687537 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.692190 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.692987 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.709479 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:37.710594 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.711452 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.727941 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:37.779019 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.782840 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.785437 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.797595 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.806857 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.815146 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:37.817055 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:37.852344 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:37.909929 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:37.914330 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:37.919214 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:37.922497 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:37.922816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.926604 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:37.930094 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:37.932897 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:37.976197 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:38.042269 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.043409 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.053440 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.053732 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.059391 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.066685 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:38.086565 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:38.104262 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.111343 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.111485 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.119109 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.120616 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.121603 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.140202 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.157389 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:38.225697 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:38.260938 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.263000 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.263437 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.264032 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.270753 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.270998 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.293500 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:38.356541 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:38.379702 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:38.399271 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.400971 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.404609 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.410648 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.432891 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.435747 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.437363 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.502761 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.503099 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.511761 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.524688 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.536243 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.538460 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.550036 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.590423 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.596176 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.603883 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.606175 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.620021 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.620573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.639286 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:38.642791 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:01:38.655015 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:38.737195 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.741520 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.744765 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.745774 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.750967 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.752404 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.769958 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:38.853147 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:38.859874 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:38.892123 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:38.896226 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:01:38.898658 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.902052 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.904289 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.906201 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.916869 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:38.921909 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.925760 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.943686 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:38.944106 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:38.944582 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:38.967428 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:38.977882 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:38.980196 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:38.999469 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:39.205966 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:39.303277 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.311617 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.319875 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:39.321717 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.333536 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.334329 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.335979 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.335986 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.351939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.362934 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.374300 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.374529 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.394665 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:39.405447 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:39.409575 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:39.430248 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:39.504405 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.506995 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.509261 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.510141 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.516827 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:39.520220 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:39.532666 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:39.533030 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:39.543781 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:01:39.561169 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:39.631261 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.632721 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.635751 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.641095 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.645284 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.660367 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:39.675501 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.755635 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.760596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.763394 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.763873 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.764281 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:39.806077 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.806795 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.839411 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.841542 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:39.841742 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:39.846482 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:39.852591 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.853235 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.854537 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.871776 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:39.966648 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:01:39.969785 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:39.973664 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:39.975502 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.978839 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:39.980154 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:39.986963 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:39.989485 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:39.990875 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:40.048049 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:01:40.051563 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:40.076162 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:01:40.107980 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240502 08:01:40.108983 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.110185 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.110543 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.110749 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.122156 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.138056 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:40.141788 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.210247 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.216707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:40.219172 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.221304 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.225776 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.230552 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.238790 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.306964 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.307754 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.308230 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.314050 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:40.330425 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:40.331617 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.339008 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:40.345258 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:40.418977 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:40.437878 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.438150 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.442178 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.444815 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:40.449792 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.452497 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.470925 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:40.503494 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:40.509919 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240502 08:01:40.529363 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:40.568174 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.570665 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.585597 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.589996 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:40.607789 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.608045 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.611270 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.616700 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.635166 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.636083 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.647938 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.658434 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.676404 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.679397 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:40.779783 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:40.799834 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.804210 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.805481 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.814837 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:40.819077 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.828059 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:40.845683 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:40.992743 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:40.992843 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:40.996599 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.005594 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.006722 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.015935 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.020534 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:41.023578 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:41.042438 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:01:41.043247 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:41.135906 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:41.137622 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.139220 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.145324 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.148216 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.148627 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.174551 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.234835 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:41.279964 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.282200 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.290338 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.290338 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.296404 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.296737 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:41.305714 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:41.330766 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.377302 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:41.380539 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.382014 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.388281 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.390308 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.399012 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.399021 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.419829 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:41.427603 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:41.431313 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:41.478425 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.481088 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.483675 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.489998 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.491269 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:41.499756 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.509097 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.565447 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:41.617764 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.621526 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.623831 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.632643 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.633498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:41.636749 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.657265 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.734978 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:41.738173 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.739208 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.746169 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.748481 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.749225 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:41.775790 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.784039 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.833603 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.834124 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.839013 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.848431 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.849828 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:41.851328 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:41.852186 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.971182 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:41.972124 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:41.976262 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:41.983520 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:41.991125 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:41.995031 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:01:41.996125 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:41.996800 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:42.014298 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:42.016793 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:42.098434 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.101727 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.107797 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:42.109513 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.127163 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.128973 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:42.140277 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:42.157683 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:42.182142 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.186483 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:42.187070 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.188306 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.190519 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.203418 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:42.203513 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:42.212822 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.318507 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.319182 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.320356 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:42.323014 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.323518 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.324440 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:42.339316 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240502 08:01:42.350132 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:42.406792 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:42.461509 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:42.461934 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:42.465525 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:42.473878 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.483008 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.488595 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:42.494396 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.494791 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.496665 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.506814 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.511355 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:42.529321 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:42.538678 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:42.539907 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:42.546751 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:42.547586 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.565811 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:42.603024 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:42.649300 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.655906 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.656669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:42.661085 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:42.670173 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.677114 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.692389 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:42.791107 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:42.870332 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.873663 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:42.874506 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.875929 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.885433 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:42.889552 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:42.893049 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:42.921206 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:01:42.993914 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:42.994665 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:42.994861 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.001570 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.004061 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.006376 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.009804 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.014707 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:43.067180 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:43.070394 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:43.123099 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.126582 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.129732 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.131500 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.135824 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.135892 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.155884 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:43.225654 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:43.260808 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.269465 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.270932 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.273656 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.278349 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.303081 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.304131 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.335156 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.339982 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.344933 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.349463 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.362824 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.366637 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.379122 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.474650 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.481102 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.483517 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.484779 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.487475 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.491057 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:43.496727 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:43.497205 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.609313 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.611224 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.611445 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.613734 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.622501 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.627240 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.649756 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.707095 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.708401 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:43.709681 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:01:43.709820 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:43.710525 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:43.714799 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.720641 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.721923 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.724718 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:43.743643 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:43.821331 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.821843 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.823392 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.825435 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.826360 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.827777 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.843883 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:43.846784 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240502 08:01:43.925621 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:43.950826 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.952031 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:43.958864 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:43.959543 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:43.960445 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:43.961143 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:43.980758 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:44.033898 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:44.077941 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.079595 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.099604 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.103746 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.105032 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:44.106762 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:44.114111 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:01:44.120204 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.122687 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.124799 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.145082 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:44.151755 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.158583 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.158912 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.174962 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:44.254709 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:44.274613 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:44.276955 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:44.277948 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.283571 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.285535 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:44.286379 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.293116 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.293176 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:44.306872 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:01:44.307968 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:44.404139 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.408224 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.411118 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:44.416443 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.420008 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.422266 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.429246 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.527196 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.527475 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.534382 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.535638 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.535949 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.536209 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.560823 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:44.665290 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:01:44.743466 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:44.744566 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.746796 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.748306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.749184 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.750128 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.752693 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.753394 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:44.838851 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:44.870011 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.875058 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:44.878793 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:44.881429 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.883770 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:44.886873 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240502 08:01:44.895108 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.907717 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.909480 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:44.924762 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:44.970152 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:44.970772 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.971206 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:44.972223 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:44.975368 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:44.983305 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:44.989715 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:44.992946 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.097666 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.098843 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.099113 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.107331 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.107784 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.108919 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.122002 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.226280 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.227842 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.235494 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.237663 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.238348 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.248646 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.261140 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.318986 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.321548 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:45.324064 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:45.324332 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.329167 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.333623 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.337704 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.338577 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:45.340006 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:45.366117 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:01:45.414045 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:45.457063 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.457862 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.459084 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.460738 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.461715 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.480239 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.485438 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:45.577044 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:45.598198 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.598650 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.611774 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.611999 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.626080 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.629770 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.634573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.645177 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.647995 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.655377 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.666038 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.668133 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.672359 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.705202 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:45.774713 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:45.801637 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.802116 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.806480 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.816162 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.819847 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.825273 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.826792 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:45.848169 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:45.898728 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:45.927358 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:45.932933 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.934393 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:45.943538 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.944226 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:45.944588 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:45.969563 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:45.981766 28246 maintenance_manager.cc:419] P 0fa72c150779467baa8c5c75f402ca60: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033392
I20240502 08:01:46.008512 28245 maintenance_manager.cc:643] P 0fa72c150779467baa8c5c75f402ca60: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.026s user 0.021s sys 0.004s Metrics: {"bytes_written":20863,"cfile_init":1,"dirs.queue_time_us":185,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1278,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":831,"lbm_writes_lt_1ms":28,"peak_mem_usage":0,"rows_written":43,"thread_start_us":170,"threads_started":2,"wal-append.queue_time_us":148}
I20240502 08:01:46.010088 28246 maintenance_manager.cc:419] P 0fa72c150779467baa8c5c75f402ca60: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 7787 bytes on disk
I20240502 08:01:46.011219 28245 maintenance_manager.cc:643] P 0fa72c150779467baa8c5c75f402ca60: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240502 08:01:46.058768 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:01:46.069092 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.069994 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.071730 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.074694 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.088575 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.089884 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:46.098960 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.143456 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.144402 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.146912 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.161439 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.162297 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:46.176609 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.180694 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:46.270648 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:46.279896 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.281940 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.285732 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.293852 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.295087 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.310385 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.317106 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:46.421795 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:46.426492 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.430321 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:46.431809 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:46.436164 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.445940 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.450678 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.454217 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:46.457582 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:46.472149 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:46.502017 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:46.515471 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240502 08:01:46.557034 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:46.618330 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.620783 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.626507 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.626636 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.627247 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.639012 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.654163 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:46.759063 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:46.774152 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.775931 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.785775 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.787921 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.795492 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.800153 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:46.809520 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.824646 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.838465 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.840544 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:46.850075 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:46.852782 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:46.854075 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.854876 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.883484 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:46.909559 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:46.972806 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.972822 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:46.974196 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:46.975863 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:46.977175 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:46.992163 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.009898 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:47.051345 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:47.072017 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:01:47.100749 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240502 08:01:47.105365 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.105830 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.111665 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.117331 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.117844 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.121416 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.129891 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.261643 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.261711 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.261940 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.266757 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.271395 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.279407 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.281236 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:47.289410 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:47.289968 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.292074 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.305339 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.318869 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.321518 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.339138 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:47.341879 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:47.359501 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:47.428685 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:47.455561 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.458775 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.458943 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.460426 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.461413 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.472518 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.482626 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:47.505402 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:47.513957 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:47.581578 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.586004 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.586272 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.588433 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.590600 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.601027 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.618315 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.707489 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.708403 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.717388 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.718669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.721354 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:47.725083 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:47.725785 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.743350 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.777930 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.782092 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.787793 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:47.790072 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:47.802927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.808279 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.813285 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:47.816962 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:47.913544 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240502 08:01:47.930429 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.935724 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:47.936589 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:47.937106 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.948882 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:47.953043 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:47.960881 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:47.972568 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:01:48.028139 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:01:48.045557 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:48.061103 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.068964 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.069567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.072356 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.072422 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.073370 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.073704 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.171298 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.174870 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.178351 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.180384 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.181546 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.182101 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.183569 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:48.225467 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:48.396723 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.397126 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.397166 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.405207 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.405923 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.412629 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.445479 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:48.450474 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:48.527127 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.531602 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.550647 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.551299 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.551693 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.558646 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.575158 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.579242 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.597380 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.599140 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.609365 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.625792 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.639050 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.646090 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:48.709846 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:48.741421 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.741547 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.745114 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.746212 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.752702 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.753321 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.765291 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.880708 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.888178 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.888448 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:48.901310 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:48.901386 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:48.903127 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.911475 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:48.970630 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:01:48.976240 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:48.986336 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:48.988235 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:48.992535 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.005129 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.014932 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.022926 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.029129 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:49.034137 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:01:49.035391 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:01:49.063525 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:49.081305 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.082300 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.082396 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.083611 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.087198 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.097543 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.100670 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.209497 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:49.213902 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:49.215487 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.216228 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.219422 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.220628 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.224622 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.224895 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:49.236337 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:01:49.271186 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:49.355000 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.370452 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.372144 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.376672 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.377321 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.381752 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.388983 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.389899 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.393059 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.405403 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.405544 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.427584 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.442263 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:49.446748 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:01:49.459349 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.548446 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.551106 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.551828 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.552835 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.554031 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.557629 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.568817 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:49.582842 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:01:49.591917 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240502 08:01:49.607026 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:49.644429 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:49.674289 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.675638 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.676288 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.680835 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.683104 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.699088 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.708578 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.805367 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.808568 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.813139 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.824898 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.827927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.847846 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.859606 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:49.867781 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:49.870756 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:49.899890 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:49.900111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.900629 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:49.901453 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:49.901685 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:49.917469 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:49.922111 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:01:49.976629 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:01:50.158464 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:50.179600 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.180105 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.188030 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.189237 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.190289 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.205425 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:50.211675 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:50.211963 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.273939 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.277534 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.280529 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.284868 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.292704 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.292709 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:50.294399 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:50.305011 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:50.407264 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.407464 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:50.407994 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:50.411317 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:50.419836 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.433491 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.442006 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.450045 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:50.536412 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:50.543306 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:50.555904 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.558813 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.559317 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.566709 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.574425 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.576742 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.582597 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.602486 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:50.607550 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.612823 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.631448 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.641494 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.641896 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:50.721544 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:50.768254 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.769547 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:50.775285 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.780521 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.780586 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.780804 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.782996 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:50.854837 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:50.882201 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:50.939090 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:50.941753 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:50.944353 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.944679 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:50.944761 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:50.970202 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:50.991797 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.009143 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.009922 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:51.011636 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.011899 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.014516 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.029713 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:51.034960 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.133477 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.134861 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:51.138455 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 57 similar messages]
I20240502 08:01:51.144335 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:51.145236 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.147656 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.158845 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.159119 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:51.206883 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:51.221697 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:51.271936 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.276927 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.277355 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:51.285323 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.286068 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(2ba62cb7ab7b429d85e0ac037c127cb4): perf score=0.033358
I20240502 08:01:51.290432 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.291824 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:51.293551 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240502 08:01:51.309952 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.349408 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(2ba62cb7ab7b429d85e0ac037c127cb4) complete. Timing: real 0.063s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":3379,"dirs.queue_time_us":10700,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":670,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":293,"threads_started":3,"wal-append.queue_time_us":178}
I20240502 08:01:51.353971 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(ab785e07fa8248e5889bea8ac755d002): perf score=0.033375
I20240502 08:01:51.354427 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(2ba62cb7ab7b429d85e0ac037c127cb4): perf score=0.033377
I20240502 08:01:51.385236 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(c65fe7f823df4b4996769fc784d3adb7): perf score=0.033383
I20240502 08:01:51.390977 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(2ba62cb7ab7b429d85e0ac037c127cb4) complete. Timing: real 0.036s user 0.025s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":12406,"dirs.queue_time_us":4212,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":5751,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":401,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":236,"threads_started":3,"wal-append.queue_time_us":382}
I20240502 08:01:51.392508 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(2ba62cb7ab7b429d85e0ac037c127cb4): 4925 bytes on disk
I20240502 08:01:51.401067 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(2ba62cb7ab7b429d85e0ac037c127cb4) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 08:01:51.403896 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(ab785e07fa8248e5889bea8ac755d002) complete. Timing: real 0.049s user 0.021s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":2038,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":603,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":115,"threads_started":1,"wal-append.queue_time_us":24900}
I20240502 08:01:51.404947 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(2ba62cb7ab7b429d85e0ac037c127cb4): 4925 bytes on disk
I20240502 08:01:51.405547 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(2ba62cb7ab7b429d85e0ac037c127cb4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:01:51.406334 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(ab785e07fa8248e5889bea8ac755d002): 4838 bytes on disk
I20240502 08:01:51.407567 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(ab785e07fa8248e5889bea8ac755d002) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240502 08:01:51.458685 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:51.460306 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:51.475476 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:51.479787 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.500494 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.501413 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.503263 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(c65fe7f823df4b4996769fc784d3adb7) complete. Timing: real 0.118s user 0.034s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":1780,"dirs.queue_time_us":17090,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":2985,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":14723,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":265,"threads_started":3,"wal-append.queue_time_us":13631}
I20240502 08:01:51.504786 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(c65fe7f823df4b4996769fc784d3adb7): 4586 bytes on disk
I20240502 08:01:51.505414 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(c65fe7f823df4b4996769fc784d3adb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 08:01:51.531733 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.585309 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:51.617518 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.637445 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.656826 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:51.657752 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.659600 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.669615 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.675073 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:51.689416 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:01:51.719985 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:51.723579 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:51.726229 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:51.746740 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:51.781734 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:51.852661 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.853605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:51.855290 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:51.857245 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.857924 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.868239 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:51.869292 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:51.903604 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(d08ad2a981fc4c13bc89d22feea7d556): perf score=0.033358
I20240502 08:01:51.910609 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(a7fd7f8583e14c91a932767ebe23b419): perf score=0.033362
I20240502 08:01:51.921926 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(d08ad2a981fc4c13bc89d22feea7d556) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":2488,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":136}
I20240502 08:01:51.922824 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(d08ad2a981fc4c13bc89d22feea7d556): 4840 bytes on disk
I20240502 08:01:51.923198 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(d08ad2a981fc4c13bc89d22feea7d556) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 08:01:51.927624 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(a7fd7f8583e14c91a932767ebe23b419) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":148,"dirs.run_wall_time_us":1377,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":165}
I20240502 08:01:51.928781 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(14bbfb3758e04aeab0df4bbab8380061): perf score=0.033365
I20240502 08:01:51.944015 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(d08ad2a981fc4c13bc89d22feea7d556): perf score=0.033373
I20240502 08:01:51.954444 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(14bbfb3758e04aeab0df4bbab8380061) complete. Timing: real 0.025s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1750,"drs_written":1,"lbm_read_time_us":156,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":6185}
I20240502 08:01:51.955533 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(a7fd7f8583e14c91a932767ebe23b419): 4925 bytes on disk
I20240502 08:01:51.955976 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(a7fd7f8583e14c91a932767ebe23b419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 08:01:51.956766 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(14bbfb3758e04aeab0df4bbab8380061): 4586 bytes on disk
I20240502 08:01:51.957149 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(14bbfb3758e04aeab0df4bbab8380061) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240502 08:01:51.988114 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:51.992435 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.000106 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(d08ad2a981fc4c13bc89d22feea7d556) complete. Timing: real 0.056s user 0.031s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":23034,"dirs.queue_time_us":7078,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":2990,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":903,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":266,"threads_started":3,"wal-append.queue_time_us":216}
I20240502 08:01:52.001562 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(d08ad2a981fc4c13bc89d22feea7d556): 4840 bytes on disk
I20240502 08:01:52.002599 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(d08ad2a981fc4c13bc89d22feea7d556) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.005123 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.010258 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:52.010675 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.011822 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(a7fd7f8583e14c91a932767ebe23b419): perf score=0.033392
I20240502 08:01:52.016640 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.026021 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:52.067255 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:52.072283 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(a7fd7f8583e14c91a932767ebe23b419) complete. Timing: real 0.060s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":195,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":544,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":454,"threads_started":2,"wal-append.queue_time_us":1812}
I20240502 08:01:52.074827 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(a7fd7f8583e14c91a932767ebe23b419): 4925 bytes on disk
I20240502 08:01:52.075426 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(a7fd7f8583e14c91a932767ebe23b419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.089093 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.098436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.099367 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.100946 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:52.101336 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.106094 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.129501 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:52.132864 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:52.160344 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:52.226575 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:52.261875 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240502 08:01:52.305418 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:52.347899 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.353201 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.356926 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.357877 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:52.359094 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.366861 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.387811 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.425494 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(c47d608895f94a2c82e695ca16677abf): perf score=0.033358
I20240502 08:01:52.445250 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.019s user 0.009s sys 0.006s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3198,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":301}
I20240502 08:01:52.446242 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf): 4586 bytes on disk
I20240502 08:01:52.446666 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.460211 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(1631d3e38209497ea2454fdfc555a319): perf score=0.033371
I20240502 08:01:52.461122 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.463222 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.468145 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.471021 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.476634 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:52.486681 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:52.500398 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:52.502419 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.042s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":11894,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":8358}
I20240502 08:01:52.503351 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319): 4925 bytes on disk
I20240502 08:01:52.503819 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.505517 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(3288b438832e488e8257227cb8934565): perf score=0.033381
I20240502 08:01:52.509116 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.521238 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.016s user 0.013s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1544,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":519,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":160}
I20240502 08:01:52.522230 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565): 4840 bytes on disk
I20240502 08:01:52.522676 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.523337 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(c47d608895f94a2c82e695ca16677abf): perf score=0.033384
I20240502 08:01:52.537945 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.014s user 0.009s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":2172,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":15872,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":172}
I20240502 08:01:52.538762 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf): 4586 bytes on disk
I20240502 08:01:52.539161 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.578132 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(1631d3e38209497ea2454fdfc555a319): perf score=0.033399
I20240502 08:01:52.607828 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.608103 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.614334 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.618984 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.041s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3606,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1748,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":569,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":217,"threads_started":2,"wal-append.queue_time_us":23882}
I20240502 08:01:52.619923 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319): 4925 bytes on disk
I20240502 08:01:52.620380 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.620694 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:52.621150 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(3288b438832e488e8257227cb8934565): perf score=0.033410
I20240502 08:01:52.631460 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.632611 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.649807 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:52.677178 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:52.682452 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.061s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":32,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1599,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":7217,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":9536}
I20240502 08:01:52.683427 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565): 4840 bytes on disk
I20240502 08:01:52.683871 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 08:01:52.739228 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.742046 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:52.749238 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:52.751117 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.752892 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.753778 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:52.762861 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.797592 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.829227 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.835227 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.836654 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.842617 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.843865 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.844117 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.853822 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:52.854562 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:52.964022 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.965569 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:52.971746 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:52.977864 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:52.979878 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.988416 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:52.999481 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.006148 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(21d939e5035541a4b003c5d276493d5c): perf score=0.033342
I20240502 08:01:53.021507 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.015s user 0.009s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":171,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1674,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":539,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":163}
I20240502 08:01:53.022574 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c): 4840 bytes on disk
I20240502 08:01:53.023052 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.042428 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(167b5f3b818048af81b1350839ebe489): perf score=0.033354
W20240502 08:01:53.046224 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:53.059445 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:53.060151 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(167b5f3b818048af81b1350839ebe489) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":230,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":1733,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":224,"threads_started":2,"wal-append.queue_time_us":366}
I20240502 08:01:53.061077 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(167b5f3b818048af81b1350839ebe489): 4925 bytes on disk
I20240502 08:01:53.061569 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(167b5f3b818048af81b1350839ebe489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.063321 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(df20642aef7748ff8962b3095e1ab56c): perf score=0.033357
I20240502 08:01:53.080302 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(df20642aef7748ff8962b3095e1ab56c) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":1712,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":521,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":279}
I20240502 08:01:53.081631 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(df20642aef7748ff8962b3095e1ab56c): 4586 bytes on disk
I20240502 08:01:53.082897 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(df20642aef7748ff8962b3095e1ab56c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240502 08:01:53.084510 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:53.086709 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:53.120760 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.125046 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:53.128679 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.134652 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.135833 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.153527 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.154385 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.157889 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:53.158471 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.174755 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.179361 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.186380 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(167b5f3b818048af81b1350839ebe489): perf score=0.033393
I20240502 08:01:53.199960 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(df20642aef7748ff8962b3095e1ab56c): perf score=0.033395
W20240502 08:01:53.208861 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:53.212683 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.235914 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.244386 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(167b5f3b818048af81b1350839ebe489) complete. Timing: real 0.058s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":17158,"dirs.run_cpu_time_us":89,"dirs.run_wall_time_us":1706,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1616,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":145,"threads_started":2,"wal-append.queue_time_us":8677}
I20240502 08:01:53.245333 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(167b5f3b818048af81b1350839ebe489): 4925 bytes on disk
I20240502 08:01:53.245887 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(167b5f3b818048af81b1350839ebe489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.246191 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.245975 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(df20642aef7748ff8962b3095e1ab56c) complete. Timing: real 0.043s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":27293,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1847,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":156,"threads_started":2,"wal-append.queue_time_us":12080}
I20240502 08:01:53.247383 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(df20642aef7748ff8962b3095e1ab56c): 4586 bytes on disk
I20240502 08:01:53.247874 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(df20642aef7748ff8962b3095e1ab56c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.337818 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:53.338224 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.338878 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.339574 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.343601 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.359630 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.368422 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.461901 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.462509 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.468626 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.469007 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:53.474416 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.477197 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.493460 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.586740 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.591401 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:53.593735 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.614238 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.614605 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.622005 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.643671 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.648964 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.656911 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.659327 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.664978 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.668272 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.669441 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:53.684281 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:53.693651 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:53.702286 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240502 08:01:53.705750 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:53.719251 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:53.748073 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(de90ee4ef90e4cb69ec37d7d9f0c86ef): perf score=0.033364
I20240502 08:01:53.750149 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(f98044a1730e4a04a7088199d4f0850e): perf score=0.033365
I20240502 08:01:53.762384 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(de90ee4ef90e4cb69ec37d7d9f0c86ef) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":163,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":160,"threads_started":2,"wal-append.queue_time_us":161}
I20240502 08:01:53.763242 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(de90ee4ef90e4cb69ec37d7d9f0c86ef): 4925 bytes on disk
I20240502 08:01:53.763631 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(de90ee4ef90e4cb69ec37d7d9f0c86ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.764340 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(fddc382ab3164ccdaa86f802c83da8be): perf score=0.033367
I20240502 08:01:53.766577 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(f98044a1730e4a04a7088199d4f0850e) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":181,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":2471,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":157,"threads_started":2,"wal-append.queue_time_us":165}
I20240502 08:01:53.767659 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(f98044a1730e4a04a7088199d4f0850e): 4840 bytes on disk
I20240502 08:01:53.768081 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(f98044a1730e4a04a7088199d4f0850e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.775946 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(fddc382ab3164ccdaa86f802c83da8be): perf score=0.033373
I20240502 08:01:53.780879 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(fddc382ab3164ccdaa86f802c83da8be) complete. Timing: real 0.016s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":69,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1528,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":98,"threads_started":1,"wal-append.queue_time_us":193}
I20240502 08:01:53.781769 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(fddc382ab3164ccdaa86f802c83da8be): 4586 bytes on disk
I20240502 08:01:53.783828 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(fddc382ab3164ccdaa86f802c83da8be) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.794170 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(fddc382ab3164ccdaa86f802c83da8be) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1518,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":429,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":2741}
I20240502 08:01:53.795210 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(fddc382ab3164ccdaa86f802c83da8be): 4586 bytes on disk
I20240502 08:01:53.795655 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(fddc382ab3164ccdaa86f802c83da8be) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240502 08:01:53.828184 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.829109 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:53.836665 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:53.840338 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.841786 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:53.842671 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:53.867341 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:53.921327 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:53.987876 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:53.991667 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.002826 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.006008 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.011137 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.013284 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.022004 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.022372 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.032656 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.033313 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.042474 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:54.062224 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:01:54.064697 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.073089 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.077097 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:54.087399 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:54.243263 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:54.247964 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:01:54.297123 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:54.298800 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(0499a3f3e6bd4069bf992e32f1abfdb0): perf score=0.033383
I20240502 08:01:54.314975 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(0499a3f3e6bd4069bf992e32f1abfdb0) complete. Timing: real 0.016s user 0.010s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":153,"dirs.run_cpu_time_us":261,"dirs.run_wall_time_us":1529,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":141}
I20240502 08:01:54.316016 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(0499a3f3e6bd4069bf992e32f1abfdb0): 4586 bytes on disk
I20240502 08:01:54.316397 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(0499a3f3e6bd4069bf992e32f1abfdb0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 08:01:54.337816 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424): perf score=0.033395
I20240502 08:01:54.352970 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.015s user 0.013s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":452,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":563,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":583}
I20240502 08:01:54.353981 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424): 4840 bytes on disk
I20240502 08:01:54.354496 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 08:01:54.402462 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.407486 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.410539 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.412626 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.422488 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.422907 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.435036 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.441407 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.441427 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.458976 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.474846 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.474851 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.487684 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.502252 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:54.556932 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:54.587966 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.592586 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.594357 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.594373 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.599210 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.601136 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.602108 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:54.615244 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:54.677490 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:01:54.705642 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:54.718400 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.722091 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.723860 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.723860 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.725013 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.734476 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.736343 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:54.745182 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240502 08:01:54.776556 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(698f494a22d34e659766b97cb24d0293): perf score=0.033354
I20240502 08:01:54.800077 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(698f494a22d34e659766b97cb24d0293) complete. Timing: real 0.023s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3570,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":2398,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":541,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":261}
I20240502 08:01:54.807240 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(698f494a22d34e659766b97cb24d0293): 4586 bytes on disk
I20240502 08:01:54.808897 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(698f494a22d34e659766b97cb24d0293) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240502 08:01:54.813746 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:54.818789 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(6a40382745f84f0eadf9e0ee2e057289): perf score=0.033360
I20240502 08:01:54.848337 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(6a40382745f84f0eadf9e0ee2e057289) complete. Timing: real 0.029s user 0.017s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":612,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":187}
I20240502 08:01:54.849570 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(6a40382745f84f0eadf9e0ee2e057289): 4925 bytes on disk
I20240502 08:01:54.850162 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(6a40382745f84f0eadf9e0ee2e057289) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 08:01:54.866722 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.868861 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.873509 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.874504 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.884209 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.885090 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.885826 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.974133 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:54.979575 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.980099 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:54.982913 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:54.985255 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:54.986446 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:54.996855 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:55.043879 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:55.095001 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.099907 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.102571 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.107968 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.111593 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.112135 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.121026 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:55.217165 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:55.227824 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:55.243878 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.245396 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:55.249128 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:55.249466 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.255705 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.267787 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.268435 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.270886 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.302237 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.305452 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.308430 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.311998 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.314740 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.338119 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.338264 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:55.371479 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:01:55.407222 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:55.442307 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.446633 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.447314 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.451077 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.455922 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.459184 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.483829 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.540318 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(9e1cd0ec72d1429f9621d217b1bf845a): perf score=0.033376
I20240502 08:01:55.561704 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.021s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3874,"dirs.run_cpu_time_us":236,"dirs.run_wall_time_us":2209,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 08:01:55.562661 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a): 4840 bytes on disk
I20240502 08:01:55.563400 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(4cb29b8b566747e5b82a8969a070123e): perf score=0.033385
I20240502 08:01:55.563886 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:01:55.585120 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(4cb29b8b566747e5b82a8969a070123e) complete. Timing: real 0.022s user 0.015s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1155,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":171,"threads_started":2,"wal-append.queue_time_us":171}
I20240502 08:01:55.586871 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.588228 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.589581 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(4cb29b8b566747e5b82a8969a070123e): 4925 bytes on disk
I20240502 08:01:55.590626 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(4cb29b8b566747e5b82a8969a070123e) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240502 08:01:55.592904 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.598996 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.599735 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.603472 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(9e1cd0ec72d1429f9621d217b1bf845a): perf score=0.033398
I20240502 08:01:55.609509 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(4cb29b8b566747e5b82a8969a070123e): perf score=0.033394
I20240502 08:01:55.622639 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.623814 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.644095 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.040s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2597,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":8123}
I20240502 08:01:55.645119 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(57f7c358ace742649df52380939ece9d): perf score=0.033408
I20240502 08:01:55.656253 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(4cb29b8b566747e5b82a8969a070123e) complete. Timing: real 0.046s user 0.015s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":4649,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":2635,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":518,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":165,"threads_started":2,"wal-append.queue_time_us":506}
I20240502 08:01:55.657399 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(4cb29b8b566747e5b82a8969a070123e): 4925 bytes on disk
I20240502 08:01:55.659250 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(4cb29b8b566747e5b82a8969a070123e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240502 08:01:55.659869 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:55.660837 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(57f7c358ace742649df52380939ece9d): perf score=0.033407
I20240502 08:01:55.661202 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4934,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":172,"threads_started":2,"wal-append.queue_time_us":817}
W20240502 08:01:55.661921 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:55.662189 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a): 4840 bytes on disk
I20240502 08:01:55.662550 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 08:01:55.663264 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d): 4586 bytes on disk
I20240502 08:01:55.663585 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240502 08:01:55.664414 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240502 08:01:55.684381 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.023s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":213,"dirs.run_wall_time_us":4180,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":605,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":3908}
I20240502 08:01:55.686326 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d): 4586 bytes on disk
I20240502 08:01:55.698817 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 08:01:55.701903 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.705511 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.706486 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.719472 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.725500 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.728229 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.734310 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:55.765569 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:55.824275 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.825184 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.827740 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.828150 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.832669 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.845677 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.868166 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.955516 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:55.957568 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.959595 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:55.970209 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:55.975739 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:55.982134 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:55.987340 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:56.023062 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:56.066211 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(07461c988fb3402c91cfd9235efbb9b1): perf score=0.033360
I20240502 08:01:56.082042 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.082283 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.091953 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.092958 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(07461c988fb3402c91cfd9235efbb9b1) complete. Timing: real 0.027s user 0.015s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":8164,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":2428,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":170}
I20240502 08:01:56.093924 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(07461c988fb3402c91cfd9235efbb9b1): 4586 bytes on disk
I20240502 08:01:56.094812 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:56.098891 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.099072 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(07461c988fb3402c91cfd9235efbb9b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4}
I20240502 08:01:56.100560 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.111079 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(07461c988fb3402c91cfd9235efbb9b1): perf score=0.033371
I20240502 08:01:56.116688 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.140311 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(07461c988fb3402c91cfd9235efbb9b1) complete. Timing: real 0.029s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":196,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1577,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":157}
I20240502 08:01:56.141158 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(dcd68a21ed7b4205a2f0cf75f80df0e6): perf score=0.033378
I20240502 08:01:56.156885 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(dcd68a21ed7b4205a2f0cf75f80df0e6) complete. Timing: real 0.016s user 0.014s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1434,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":620,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":181}
I20240502 08:01:56.157850 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(dcd68a21ed7b4205a2f0cf75f80df0e6): 4836 bytes on disk
W20240502 08:01:56.158125 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:01:56.158329 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(dcd68a21ed7b4205a2f0cf75f80df0e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:01:56.159165 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(07461c988fb3402c91cfd9235efbb9b1): 4586 bytes on disk
I20240502 08:01:56.159540 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(07461c988fb3402c91cfd9235efbb9b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240502 08:01:56.165683 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(dcd68a21ed7b4205a2f0cf75f80df0e6): perf score=0.033382
I20240502 08:01:56.178014 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(dcd68a21ed7b4205a2f0cf75f80df0e6) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":184,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1478,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":447,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":163}
I20240502 08:01:56.178777 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(dcd68a21ed7b4205a2f0cf75f80df0e6): 4836 bytes on disk
I20240502 08:01:56.179180 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(dcd68a21ed7b4205a2f0cf75f80df0e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240502 08:01:56.202138 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:56.291157 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.296223 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:56.299001 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.302852 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.306677 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.315713 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.318996 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:56.389884 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:56.390093 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:01:56.399680 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:56.442442 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.449604 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.456704 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.457121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.462906 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.474407 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:56.488397 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:01:56.489624 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:56.493445 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.495582 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.501797 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.504437 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:56.515937 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.520509 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.525552 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.540808 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:56.618633 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:56.641129 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.642788 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.645749 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.646086 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.653623 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.666901 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:56.667132 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.702976 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(3598dbda342f49aea2384b5beb17d7a1): perf score=0.033364
I20240502 08:01:56.717821 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(3598dbda342f49aea2384b5beb17d7a1) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":182,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":1621,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":191}
I20240502 08:01:56.718786 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(3598dbda342f49aea2384b5beb17d7a1): 4840 bytes on disk
I20240502 08:01:56.719273 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(3598dbda342f49aea2384b5beb17d7a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240502 08:01:56.790447 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.797720 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.799340 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.802467 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.804355 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:56.807686 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:56.838150 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:01:56.845932 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.852231 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(7dbf7921ae84476ab53fbd4707f463eb): perf score=0.033397
I20240502 08:01:56.870563 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(7dbf7921ae84476ab53fbd4707f463eb) complete. Timing: real 0.018s user 0.014s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":169,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1799,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":628,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":166,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 08:01:56.871611 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(7dbf7921ae84476ab53fbd4707f463eb): 4584 bytes on disk
I20240502 08:01:56.872328 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(7dbf7921ae84476ab53fbd4707f463eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
W20240502 08:01:56.914984 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:56.948582 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:56.950482 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:56.950973 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:56.951488 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:56.957805 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.967692 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:56.994524 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:56.996712 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:01:57.003530 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:57.029489 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:57.032400 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.034891 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.042526 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.043221 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:57.048652 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.068421 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:57.121909 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:57.157980 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:57.167992 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:57.173754 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.175350 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.177675 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.179148 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.185019 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.305999 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.309525 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:57.313961 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.331025 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.332821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.334144 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:57.339541 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240502 08:01:57.348428 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:57.349706 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.359591 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.359707 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.361241 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.364761 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.370623 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.375595 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:57.379262 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:57.481884 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:57.500063 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.501875 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:57.502758 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.503356 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:57.506372 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:57.506546 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.513039 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.540288 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.632017 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.632017 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:57.634456 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.636359 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.643788 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.652217 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.653882 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:01:57.680258 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:01:57.697389 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:57.757404 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:57.758705 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.764182 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:57.765048 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.771513 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.785676 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:57.793318 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:57.802819 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.824168 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:57.826622 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.829180 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:57.846853 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:57.855270 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.872395 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:57.872857 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:58.033102 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:01:58.042407 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:01:58.120491 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240502 08:01:58.126396 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:58.133698 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.133965 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.137128 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:58.149458 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.155819 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.160490 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.182299 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:58.215724 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:58.226641 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.228148 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.230706 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.234930 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.243705 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:58.244226 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.274396 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:58.333690 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:01:58.370822 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.371253 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.373438 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.379047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:58.380884 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.395781 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.410534 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:58.453598 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:58.506785 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.511552 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:58.516595 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.520292 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.526168 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.527137 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:58.558087 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.560317 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.565009 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.565371 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:58.575721 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.576155 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.576787 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.599669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:01:58.651484 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:01:58.685678 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:58.698429 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:58.703084 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.707482 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:58.711349 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.712390 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.716629 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.729552 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.834028 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.835741 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.836894 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.846616 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.847021 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:58.853555 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:58.854399 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:58.896472 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:58.963626 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:58.969076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:58.969574 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:58.976953 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:58.990190 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.002928 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.014139 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.022806 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.023260 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.030783 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.031615 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.038667 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.050668 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:59.051514 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:01:59.113258 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:01:59.130039 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:59.154950 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.156566 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.174729 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.179692 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.185693 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.185698 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:01:59.209481 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:59.217655 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:59.291822 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:59.302711 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.305696 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.312841 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.321691 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.324404 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:59.325068 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.336628 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:59.346421 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:01:59.379103 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:01:59.421612 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:01:59.436900 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.439962 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.440229 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.452687 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:59.459444 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.461555 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.466846 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:59.505790 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240502 08:01:59.523752 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.524384 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.526263 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.530987 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.532680 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:59.544180 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.547700 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:59.604813 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:59.661844 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.661978 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.666059 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.667105 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.669724 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.676177 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:59.710549 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:01:59.723101 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:01:59.757265 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:01:59.787249 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:01:59.789335 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.799244 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:01:59.809063 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:01:59.811661 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:01:59.812882 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:01:59.823267 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.002694 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.012245 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.014931 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.015231 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.022825 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.024564 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.046049 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:00.126399 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:00.138584 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:00.140128 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.141789 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:00.142807 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:00.147034 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.149909 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.152695 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.160146 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.173359 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:00.183071 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:02:00.189047 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:00.225271 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.227174 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.231657 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.232774 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.242414 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.257167 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.258927 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:00.319326 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:00.356146 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.358206 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.360960 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.363803 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.366881 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.370409 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.373745 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:00.390349 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:00.486515 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.495162 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.495301 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.498261 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.515528 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:00.516197 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:00.516610 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:00.517024 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:00.521234 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.532639 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.534924 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.540199 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.551250 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.564702 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.584775 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.595687 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.650684 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(d08ad2a981fc4c13bc89d22feea7d556): perf score=0.033336
W20240502 08:02:00.698189 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:00.708990 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.710407 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.713860 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(d08ad2a981fc4c13bc89d22feea7d556) complete. Timing: real 0.063s user 0.031s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":158,"dirs.queue_time_us":214,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":1670,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":699,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":255,"threads_started":3,"wal-append.queue_time_us":25846}
I20240502 08:02:00.717381 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(d08ad2a981fc4c13bc89d22feea7d556): 4840 bytes on disk
I20240502 08:02:00.718173 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(d08ad2a981fc4c13bc89d22feea7d556) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":212,"lbm_reads_lt_1ms":4}
I20240502 08:02:00.719033 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(7dbf7921ae84476ab53fbd4707f463eb): perf score=0.033346
I20240502 08:02:00.721369 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.722601 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.742858 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.743220 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.752002 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.772794 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(7dbf7921ae84476ab53fbd4707f463eb) complete. Timing: real 0.054s user 0.019s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1475,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":688,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":13302}
I20240502 08:02:00.773636 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(7dbf7921ae84476ab53fbd4707f463eb): 4584 bytes on disk
I20240502 08:02:00.774966 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(7dbf7921ae84476ab53fbd4707f463eb) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":555,"lbm_reads_lt_1ms":4}
W20240502 08:02:00.809808 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:00.884574 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.889149 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:00.900043 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:00.901266 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:00.902724 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.904834 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:00.929328 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424): perf score=0.033357
I20240502 08:02:00.935343 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:00.984834 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.055s user 0.030s sys 0.005s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":40466,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1184,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":227,"threads_started":3,"wal-append.queue_time_us":132}
I20240502 08:02:00.986356 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424): 4840 bytes on disk
I20240502 08:02:00.986922 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.029028 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(9e1cd0ec72d1429f9621d217b1bf845a): perf score=0.033392
I20240502 08:02:01.033138 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.052924 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.054574 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.058400 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.060901 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.062242 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.033s user 0.017s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":2735,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2571,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":170}
I20240502 08:02:01.063103 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a): 4840 bytes on disk
I20240502 08:02:01.073511 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.075023 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.077770 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:01.096391 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240502 08:02:01.131223 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:01.135371 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:01.155112 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.156083 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.164955 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.176000 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.185011 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.201916 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.202409 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.238502 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(5d1c1fa046b741aa8cf296aa57847fe3): perf score=0.033368
W20240502 08:02:01.251843 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240502 08:02:01.260483 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(5d1c1fa046b741aa8cf296aa57847fe3) complete. Timing: real 0.022s user 0.019s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1590,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":80,"threads_started":1,"wal-append.queue_time_us":174}
I20240502 08:02:01.261250 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(5d1c1fa046b741aa8cf296aa57847fe3): 4840 bytes on disk
I20240502 08:02:01.261703 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(5d1c1fa046b741aa8cf296aa57847fe3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.320322 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.325210 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(1631d3e38209497ea2454fdfc555a319): perf score=0.033384
I20240502 08:02:01.332731 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.332861 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.333492 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.337990 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.348942 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.358677 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.033s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":61,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1511,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":604,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":187}
I20240502 08:02:01.359686 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319): 4925 bytes on disk
I20240502 08:02:01.360189 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.360644 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.360929 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(c47d608895f94a2c82e695ca16677abf): perf score=0.033387
W20240502 08:02:01.380986 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:01.395386 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.034s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":216,"dirs.run_cpu_time_us":161,"dirs.run_wall_time_us":1620,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":111,"threads_started":1,"wal-append.queue_time_us":17602}
I20240502 08:02:01.396541 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(a28d406b43194bc6b775707d021618f0): perf score=0.033393
I20240502 08:02:01.418332 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(a28d406b43194bc6b775707d021618f0) complete. Timing: real 0.022s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":2076,"drs_written":1,"lbm_read_time_us":1157,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":121,"threads_started":1,"wal-append.queue_time_us":242}
I20240502 08:02:01.419401 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(a28d406b43194bc6b775707d021618f0): 4925 bytes on disk
I20240502 08:02:01.419911 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(a28d406b43194bc6b775707d021618f0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.420760 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(a7fd7f8583e14c91a932767ebe23b419): perf score=0.033382
I20240502 08:02:01.442413 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(a7fd7f8583e14c91a932767ebe23b419) complete. Timing: real 0.022s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":246,"dirs.run_wall_time_us":1617,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2097,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":96,"threads_started":1,"wal-append.queue_time_us":384}
I20240502 08:02:01.443451 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(a7fd7f8583e14c91a932767ebe23b419): 4925 bytes on disk
I20240502 08:02:01.443933 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(a7fd7f8583e14c91a932767ebe23b419) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.444674 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf): 4586 bytes on disk
I20240502 08:02:01.445863 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":867,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.471130 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(afc9c4d9dd5b4c039686ccd918616859): perf score=0.033392
I20240502 08:02:01.489181 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.491408 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.492700 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.494643 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.506861 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.513341 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(afc9c4d9dd5b4c039686ccd918616859): perf score=0.033393
I20240502 08:02:01.525066 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.525369 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.561952 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(afc9c4d9dd5b4c039686ccd918616859) complete. Timing: real 0.090s user 0.034s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":71707,"drs_written":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1778,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":291,"threads_started":3,"wal-append.queue_time_us":297}
I20240502 08:02:01.569623 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(04d358245cdf4b2ebb0b5005f75cf1e0): perf score=0.033404
I20240502 08:02:01.574165 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.580804 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.584158 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.584642 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.600926 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.601202 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.616648 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.638399 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(04d358245cdf4b2ebb0b5005f75cf1e0) complete. Timing: real 0.067s user 0.020s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1584,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":685,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":1,"wal-append.queue_time_us":27410}
I20240502 08:02:01.639452 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(04d358245cdf4b2ebb0b5005f75cf1e0): 4925 bytes on disk
I20240502 08:02:01.640693 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(04d358245cdf4b2ebb0b5005f75cf1e0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.641623 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(5d1c1fa046b741aa8cf296aa57847fe3): perf score=0.033422
I20240502 08:02:01.653892 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(afc9c4d9dd5b4c039686ccd918616859) complete. Timing: real 0.140s user 0.018s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":251,"dirs.run_wall_time_us":1751,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":566,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":10087}
I20240502 08:02:01.654901 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(afc9c4d9dd5b4c039686ccd918616859): 4925 bytes on disk
I20240502 08:02:01.655450 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(afc9c4d9dd5b4c039686ccd918616859) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.656268 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(de90ee4ef90e4cb69ec37d7d9f0c86ef): perf score=0.033416
W20240502 08:02:01.664829 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:01.665311 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(5d1c1fa046b741aa8cf296aa57847fe3) complete. Timing: real 0.024s user 0.021s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1424,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":665,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":515}
I20240502 08:02:01.666563 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(c47d608895f94a2c82e695ca16677abf): perf score=0.033398
W20240502 08:02:01.666673 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:02:01.670192 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:01.678505 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(de90ee4ef90e4cb69ec37d7d9f0c86ef) complete. Timing: real 0.022s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1724,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":123,"threads_started":1,"wal-append.queue_time_us":224}
I20240502 08:02:01.679497 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(de90ee4ef90e4cb69ec37d7d9f0c86ef): 4925 bytes on disk
I20240502 08:02:01.679924 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(de90ee4ef90e4cb69ec37d7d9f0c86ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.681514 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(1631d3e38209497ea2454fdfc555a319): perf score=0.033387
I20240502 08:02:01.691356 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.024s user 0.015s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1515,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":944,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":163}
I20240502 08:02:01.692433 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(698f494a22d34e659766b97cb24d0293): perf score=0.033405
I20240502 08:02:01.697516 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(f98044a1730e4a04a7088199d4f0850e): perf score=0.033389
I20240502 08:02:01.709937 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.028s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1441,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":609,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":131,"threads_started":1,"wal-append.queue_time_us":214}
I20240502 08:02:01.710883 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319): 4925 bytes on disk
I20240502 08:02:01.711284 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(1631d3e38209497ea2454fdfc555a319) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.712011 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(07461c988fb3402c91cfd9235efbb9b1): perf score=0.033338
I20240502 08:02:01.734555 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(698f494a22d34e659766b97cb24d0293) complete. Timing: real 0.042s user 0.018s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3223,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1239,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":6733,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":635,"thread_start_us":124,"threads_started":1,"wal-append.queue_time_us":240}
I20240502 08:02:01.735313 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(f98044a1730e4a04a7088199d4f0850e) complete. Timing: real 0.038s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1774,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":11963,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":155}
I20240502 08:02:01.736310 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(afc9c4d9dd5b4c039686ccd918616859): 4925 bytes on disk
I20240502 08:02:01.736816 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(afc9c4d9dd5b4c039686ccd918616859) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.737104 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(f98044a1730e4a04a7088199d4f0850e): 4840 bytes on disk
I20240502 08:02:01.737826 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(5d1c1fa046b741aa8cf296aa57847fe3): 4840 bytes on disk
I20240502 08:02:01.738266 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(f98044a1730e4a04a7088199d4f0850e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.738955 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(5d1c1fa046b741aa8cf296aa57847fe3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.739220 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(07461c988fb3402c91cfd9235efbb9b1) complete. Timing: real 0.027s user 0.015s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":99,"dirs.run_cpu_time_us":256,"dirs.run_wall_time_us":1692,"drs_written":1,"lbm_read_time_us":1446,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":124,"threads_started":1,"wal-append.queue_time_us":7605}
I20240502 08:02:01.740088 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(07461c988fb3402c91cfd9235efbb9b1): 4586 bytes on disk
I20240502 08:02:01.740315 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(0499a3f3e6bd4069bf992e32f1abfdb0): perf score=0.033414
I20240502 08:02:01.740792 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(07461c988fb3402c91cfd9235efbb9b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 08:02:01.742769 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:02:01.754370 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:02:01.754786 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:01.768817 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(0499a3f3e6bd4069bf992e32f1abfdb0) complete. Timing: real 0.028s user 0.019s sys 0.001s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":176,"lbm_reads_lt_1ms":4,"lbm_write_time_us":571,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":1,"wal-append.queue_time_us":2119}
I20240502 08:02:01.770042 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(de90ee4ef90e4cb69ec37d7d9f0c86ef): perf score=0.033421
I20240502 08:02:01.794569 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(de90ee4ef90e4cb69ec37d7d9f0c86ef) complete. Timing: real 0.024s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1533,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":597,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":125,"threads_started":1,"wal-append.queue_time_us":4695}
I20240502 08:02:01.795678 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(de90ee4ef90e4cb69ec37d7d9f0c86ef): 4925 bytes on disk
I20240502 08:02:01.796135 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(de90ee4ef90e4cb69ec37d7d9f0c86ef) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.797016 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(0499a3f3e6bd4069bf992e32f1abfdb0): 4586 bytes on disk
I20240502 08:02:01.797403 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(0499a3f3e6bd4069bf992e32f1abfdb0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.798193 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(698f494a22d34e659766b97cb24d0293): 4586 bytes on disk
I20240502 08:02:01.798563 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(698f494a22d34e659766b97cb24d0293) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.799265 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf): 4586 bytes on disk
I20240502 08:02:01.799651 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(c47d608895f94a2c82e695ca16677abf) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4}
I20240502 08:02:01.810842 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.823482 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.824859 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.826833 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.827505 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.839211 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.853058 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:01.909586 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:02:01.939509 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:02:01.942314 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.944808 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:01.946128 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:01.947137 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240502 08:02:01.951294 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:01.951566 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:01.953864 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:01.972349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:01.989711 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(04d358245cdf4b2ebb0b5005f75cf1e0): perf score=0.033385
I20240502 08:02:02.007043 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(04d358245cdf4b2ebb0b5005f75cf1e0) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":1537,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":596,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":156}
I20240502 08:02:02.007814 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(04d358245cdf4b2ebb0b5005f75cf1e0): 4925 bytes on disk
I20240502 08:02:02.008200 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(04d358245cdf4b2ebb0b5005f75cf1e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.051355 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(14bbfb3758e04aeab0df4bbab8380061): perf score=0.033361
I20240502 08:02:02.063083 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(14bbfb3758e04aeab0df4bbab8380061) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1577,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":464,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":164}
I20240502 08:02:02.064132 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(2ba62cb7ab7b429d85e0ac037c127cb4): perf score=0.033339
I20240502 08:02:02.077389 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(2ba62cb7ab7b429d85e0ac037c127cb4) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1589,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":508,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":119,"threads_started":1,"wal-append.queue_time_us":217}
I20240502 08:02:02.078203 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(2ba62cb7ab7b429d85e0ac037c127cb4): 4925 bytes on disk
I20240502 08:02:02.078621 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(2ba62cb7ab7b429d85e0ac037c127cb4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.079242 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(14bbfb3758e04aeab0df4bbab8380061): 4586 bytes on disk
I20240502 08:02:02.079658 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(14bbfb3758e04aeab0df4bbab8380061) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240502 08:02:02.145666 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:02.183089 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.183600 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.184670 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.185097 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.192313 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.192653 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:02.197346 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:02.201614 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:02.205893 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:02.243089 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(14bbfb3758e04aeab0df4bbab8380061): perf score=0.033380
I20240502 08:02:02.261158 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(14bbfb3758e04aeab0df4bbab8380061) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":692,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":4576,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":745,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":136,"threads_started":2,"wal-append.queue_time_us":145}
I20240502 08:02:02.262100 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(ab785e07fa8248e5889bea8ac755d002): perf score=0.033360
I20240502 08:02:02.279805 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(ab785e07fa8248e5889bea8ac755d002) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":104,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1518,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":85,"threads_started":1,"wal-append.queue_time_us":175}
I20240502 08:02:02.280903 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(ab785e07fa8248e5889bea8ac755d002): 4838 bytes on disk
I20240502 08:02:02.281313 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(ab785e07fa8248e5889bea8ac755d002) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.282203 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(14bbfb3758e04aeab0df4bbab8380061): 4586 bytes on disk
I20240502 08:02:02.282711 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(14bbfb3758e04aeab0df4bbab8380061) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.324364 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.327100 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:02.329991 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.335198 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.336629 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:02.349090 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.365507 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.413182 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.413867 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.421681 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.421681 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.426028 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:02.430418 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:02.461889 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.510201 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(21d939e5035541a4b003c5d276493d5c): perf score=0.033392
I20240502 08:02:02.527025 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.017s user 0.016s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":192,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1776,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":507}
I20240502 08:02:02.528640 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(4cb29b8b566747e5b82a8969a070123e): perf score=0.033369
I20240502 08:02:02.546377 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(4cb29b8b566747e5b82a8969a070123e) complete. Timing: real 0.018s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1666,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":658,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":92,"threads_started":1,"wal-append.queue_time_us":180}
I20240502 08:02:02.547629 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(4cb29b8b566747e5b82a8969a070123e): 4925 bytes on disk
I20240502 08:02:02.551339 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(4cb29b8b566747e5b82a8969a070123e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.552091 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c): 4840 bytes on disk
I20240502 08:02:02.553534 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":457,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.559434 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.559887 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:02.563316 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.563514 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:02.564165 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.577625 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.582074 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(dcd68a21ed7b4205a2f0cf75f80df0e6): perf score=0.033345
I20240502 08:02:02.594187 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:02.610862 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:02.615551 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(dcd68a21ed7b4205a2f0cf75f80df0e6) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2307,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":676,"thread_start_us":174,"threads_started":2,"wal-append.queue_time_us":165}
I20240502 08:02:02.616518 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(df20642aef7748ff8962b3095e1ab56c): perf score=0.033344
I20240502 08:02:02.632850 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(df20642aef7748ff8962b3095e1ab56c) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1418,"drs_written":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4,"lbm_write_time_us":617,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":377}
I20240502 08:02:02.634512 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(dcd68a21ed7b4205a2f0cf75f80df0e6): 4836 bytes on disk
I20240502 08:02:02.635033 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(dcd68a21ed7b4205a2f0cf75f80df0e6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.636261 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(df20642aef7748ff8962b3095e1ab56c): 4586 bytes on disk
I20240502 08:02:02.636679 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(df20642aef7748ff8962b3095e1ab56c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:02:02.700412 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.705756 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.715461 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.718763 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.719777 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:02.733763 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:02.734720 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:02.739078 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:02.751375 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:02.788708 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.790133 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.790392 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:02.795243 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.804299 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.805716 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(6a40382745f84f0eadf9e0ee2e057289): perf score=0.033396
I20240502 08:02:02.838876 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:02.841847 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.843078 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(6a40382745f84f0eadf9e0ee2e057289) complete. Timing: real 0.037s user 0.017s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":1614,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":2303}
I20240502 08:02:02.844009 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(6a40382745f84f0eadf9e0ee2e057289): 4925 bytes on disk
W20240502 08:02:02.844310 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:02.844655 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(6a40382745f84f0eadf9e0ee2e057289) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240502 08:02:02.852942 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:02:02.891088 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:02.936648 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.937564 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:02.937712 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:02.947142 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:02.948797 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:02.961380 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:02.985380 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:03.009570 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:03.084030 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.088007 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.095541 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.098232 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.098990 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.108038 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.122524 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:03.140504 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:03.154790 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.156651 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.161998 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.166630 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.167791 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.188776 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.204821 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:03.261305 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:03.289022 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.289472 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.292627 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.294258 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.299057 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.302395 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.315685 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.415308 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.415366 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.424433 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.424643 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.428426 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.430965 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.435490 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:03.518939 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:03.559019 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.570624 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.573874 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.574149 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.578492 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.579872 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.596740 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.598349 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.603858 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.605201 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.608587 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.612537 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.622442 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.627815 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:03.682133 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:03.737565 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:03.761981 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.762782 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.765691 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:03.770350 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.791965 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:03.792382 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:03.792574 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:03.801071 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:03.890038 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:03.949621 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:03.950785 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:03.951411 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:03.957727 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240502 08:02:03.967218 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.967607 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:03.970544 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:03.979377 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:03.981029 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:04.092933 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:04.097765 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.101300 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.103688 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.109308 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.111804 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.112156 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.124841 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:04.176036 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:04.227130 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.230834 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.232127 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:04.232844 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:04.243876 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.244632 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.245748 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.247277 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:04.309463 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:04.345695 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.346650 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.357787 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.358031 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.359026 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.365520 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:04.384562 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.424075 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.426057 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.430763 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.436689 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.440383 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.441275 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:04.445065 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:04.460530 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240502 08:02:04.469664 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:04.566268 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.566535 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.568346 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.571146 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:04.572681 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.576807 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:04.581966 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:04.583959 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:04.688135 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:02:04.715835 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.718459 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.725605 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.726672 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.732664 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.733531 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.737527 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:04.774674 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.780282 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.780565 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:04.793962 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:04.795439 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.797534 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.805578 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.807546 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.905606 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.905920 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:04.908349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:04.909011 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:04.911759 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:04.916419 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:04.946713 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.043000 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:05.043799 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:05.047120 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.048071 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:05.057170 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:05.057896 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.069474 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.072783 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:05.121840 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:05.177825 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:05.178267 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.188656 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:05.188941 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.208138 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.217839 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.217926 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:05.231495 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:05.238556 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.250249 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.266239 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.271878 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:05.276243 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.276773 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:05.293951 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:05.325605 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:02:05.386498 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:05.396855 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:05.398433 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:05.400422 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.403581 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.414572 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.424221 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:05.424698 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:05.511693 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:02:05.513720 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:02:05.521807 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:05.532975 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:05.538602 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:05.540097 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.542410 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.546538 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:05.553977 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:05.566874 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.580343 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.611521 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:05.623762 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.624500 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.627348 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:05.634795 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.640235 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:05.644376 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:05.723304 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:05.894188 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.899775 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:05.905076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:05.909435 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:05.914589 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:05.914906 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:05.923661 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:05.958724 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:06.014328 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.014744 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.020757 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.021407 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.031566 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.031566 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.040351 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.096709 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.100571 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.102721 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.105333 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.108012 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.109370 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.123139 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:06.133399 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:06.219954 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.221062 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.231621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.234072 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.235517 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.251174 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.257113 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:06.340166 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:06.340653 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:06.362680 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.367192 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.379055 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:06.382764 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:06.388139 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.390338 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.401527 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.408277 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.412894 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.413465 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:06.422542 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:02:06.424787 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.428371 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.428810 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.433763 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.449436 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:06.461328 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:02:06.530215 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:06.563321 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.565757 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.571636 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.573755 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.573946 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.583973 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.585486 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:06.601764 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240502 08:02:06.674078 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:06.693569 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.698276 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.700744 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.701045 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.708402 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.712095 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:06.728024 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240502 08:02:06.738902 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:06.741998 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:06.824831 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.831322 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.834400 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:06.843513 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:02:06.845985 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.846515 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.854538 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:06.865326 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:06.869988 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:06.870342 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.894644 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:06.897138 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.916672 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:06.917529 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:06.917706 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:06.921329 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:06.939399 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:07.026791 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 08:02:07.032110 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.032507 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.033640 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.034206 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.042946 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.051941 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.059799 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:07.077654 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:07.158747 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.162148 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.169610 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:07.170783 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.171541 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.176676 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.183375 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.261029 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.268191 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.268409 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.281409 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:07.286018 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.297772 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.311116 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:07.324615 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:07.365705 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.369810 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.370249 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:07.372498 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.376655 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.390357 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.391121 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:07.559846 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:07.601644 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.603780 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.606390 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.615136 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.616086 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.623519 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:07.624830 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:07.652685 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:02:07.689421 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:07.717411 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.717832 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.723685 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:07.731621 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.732223 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.732712 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.735108 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.847275 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:07.847337 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.848508 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.849901 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.867281 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.871591 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.885337 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:07.930428 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:07.969188 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:07.970125 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:07.979105 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:07.979626 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:07.993966 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.007076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.009150 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.034066 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.036203 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.045276 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.055442 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:08.057797 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:08.058857 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.060294 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.066020 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:08.103823 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240502 08:02:08.107761 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:08.116072 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:08.154879 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424): perf score=0.033382
I20240502 08:02:08.175386 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.019s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4565,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":167,"threads_started":2,"wal-append.queue_time_us":310}
I20240502 08:02:08.176338 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424): 4840 bytes on disk
I20240502 08:02:08.176776 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 08:02:08.190043 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.191673 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.192512 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.197073 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.214673 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.215507 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.235247 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:08.327399 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:08.339083 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.342705 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.344497 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.349474 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.352423 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.365248 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.367062 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.435997 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.450589 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.453823 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.453931 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.455372 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.466085 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.483189 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:08.511700 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:08.531462 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.540629 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.542155 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.544042 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.546761 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.552649 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.564474 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.658432 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.661326 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.664930 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.669103 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.669437 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.670526 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.673365 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:08.690560 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:08.727069 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:08.804064 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.813162 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.821334 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.822140 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.823868 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.828168 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.842845 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.844439 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.848616 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.865453 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:08.865845 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:08.869313 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.872277 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:08.881261 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:08.908713 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:02:08.919006 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:08.996738 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:08.997888 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:08.998807 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.000803 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.008821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:09.009583 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:09.011691 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:09.034358 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240502 08:02:09.035205 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:09.062371 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:09.128084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:09.128217 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:09.130669 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.132404 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.143699 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.146427 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:09.151922 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:09.152632 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.227733 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.229236 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:09.229319 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.229833 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:09.254266 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.256829 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.264983 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:09.296331 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:02:09.432057 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:09.477191 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:09.478273 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.485492 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.485533 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:09.486212 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.504096 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.508392 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.511904 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(21d939e5035541a4b003c5d276493d5c): perf score=0.033337
I20240502 08:02:09.537485 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.021s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1653,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":135,"threads_started":2,"wal-append.queue_time_us":138}
I20240502 08:02:09.538702 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c): 4840 bytes on disk
I20240502 08:02:09.539166 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240502 08:02:09.552145 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:09.613237 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.615214 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.626914 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.628196 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:09.631358 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.649300 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.662040 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:09.673345 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:09.682281 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:09.684762 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.698843 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.706001 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.707623 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.708034 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.720065 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:09.762320 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:09.803339 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:02:09.828477 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:09.829988 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:09.832062 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.834280 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.843659 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.847540 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.848191 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:09.971341 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:09.976276 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:09.977905 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:09.986608 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:09.993633 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:09.994271 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:10.008694 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:10.009969 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.024142 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.039053 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.045939 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.048902 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.049932 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:10.050057 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.056777 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.061110 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.157383 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.158169 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:10.164299 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:10.168041 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:10.171188 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.174183 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.186707 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.187693 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:10.247140 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:10.297832 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.297937 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.298401 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.305681 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:10.308060 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.320230 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.320652 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:10.371163 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:10.430424 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.441768 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.459961 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.463760 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.463830 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.464121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.472095 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.472260 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.472436 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.475469 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.482904 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:10.499543 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.516103 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:10.516544 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:10.524152 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.613106 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.615660 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.624742 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.629446 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:10.632298 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.633898 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.633994 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:10.698426 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:02:10.704774 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240502 08:02:10.728106 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:02:10.729173 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:10.746349 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.748575 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.749121 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.758189 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.758647 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.759250 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.770628 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:10.862481 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:10.876629 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.879272 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.880703 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.893716 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:10.897524 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.920166 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.931737 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.948628 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:10.948823 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:10.950850 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.958797 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:10.968755 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:10.974933 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:10.982959 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:10.987087 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:11.075930 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:11.268661 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.271065 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:11.280797 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.283063 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:11.293102 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:11.304639 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.306365 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:11.334518 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:11.370764 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:11.371433 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.372228 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:11.373555 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:11.373800 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.375813 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:11.382369 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.389955 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:11.401866 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:02:11.450537 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240502 08:02:11.451249 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:11.499284 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.499773 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.501029 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.503700 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:11.508651 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:11.511408 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:11.524078 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:11.547721 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:11.632531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.633476 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.639047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:11.654762 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:11.656508 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.659183 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.659238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.670437 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:11.675829 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:11.701253 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240502 08:02:11.702476 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.704640 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:11.708408 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.737800 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.740597 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:11.776908 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:11.780197 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:11.817111 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(0499a3f3e6bd4069bf992e32f1abfdb0): perf score=0.033334
I20240502 08:02:11.833719 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.836278 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:11.841903 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.850392 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:11.851038 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.854751 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:11.858886 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(0499a3f3e6bd4069bf992e32f1abfdb0) complete. Timing: real 0.042s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":11293,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":4427,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":146,"threads_started":2,"wal-append.queue_time_us":17772}
I20240502 08:02:11.861251 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(0499a3f3e6bd4069bf992e32f1abfdb0): 4586 bytes on disk
I20240502 08:02:11.861985 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(0499a3f3e6bd4069bf992e32f1abfdb0) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240502 08:02:11.862246 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:11.939831 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(a28d406b43194bc6b775707d021618f0): perf score=0.033392
I20240502 08:02:11.957787 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(a28d406b43194bc6b775707d021618f0) complete. Timing: real 0.018s user 0.015s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":3349,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":520,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":3840,"thread_start_us":190,"threads_started":2,"wal-append.queue_time_us":396}
I20240502 08:02:11.958810 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(a28d406b43194bc6b775707d021618f0): 4925 bytes on disk
I20240502 08:02:11.959285 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(a28d406b43194bc6b775707d021618f0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240502 08:02:11.983992 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:11.987243 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.987640 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:11.988260 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:11.990682 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:11.993953 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 08:02:11.996635 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:12.020531 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:12.024653 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:12.066601 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:12.082787 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:02:12.087215 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.088718 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.089780 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.092808 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.115425 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.117138 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.131995 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.166399 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.167608 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.175381 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.180938 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.184103 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.188342 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.206650 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:12.252087 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:12.260663 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:12.307996 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.310129 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.312271 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.312685 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.313314 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.335819 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.351152 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.449573 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.453795 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.463976 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling FlushMRSOp(c65fe7f823df4b4996769fc784d3adb7): perf score=0.033368
W20240502 08:02:12.470454 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:12.471752 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.475776 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.486425 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.491302 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.495354 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.498513 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.500994 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.511629 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.513036 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.530418 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.550617 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:12.556393 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:12.558501 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.562395 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: FlushMRSOp(c65fe7f823df4b4996769fc784d3adb7) complete. Timing: real 0.098s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":3095,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":2167,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":593,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":5791}
I20240502 08:02:12.563647 30262 maintenance_manager.cc:419] P 6c68c916f39a45f697df7eaf8973225a: Scheduling UndoDeltaBlockGCOp(c65fe7f823df4b4996769fc784d3adb7): 4586 bytes on disk
I20240502 08:02:12.564188 30152 maintenance_manager.cc:643] P 6c68c916f39a45f697df7eaf8973225a: UndoDeltaBlockGCOp(c65fe7f823df4b4996769fc784d3adb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240502 08:02:12.564522 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:12.650789 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.655738 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.658303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.660619 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.661427 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.664732 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.679214 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:12.693384 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:12.720165 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(3598dbda342f49aea2384b5beb17d7a1): perf score=0.033360
I20240502 08:02:12.742945 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(3598dbda342f49aea2384b5beb17d7a1) complete. Timing: real 0.023s user 0.012s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1561,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":675,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":153,"threads_started":2,"wal-append.queue_time_us":162}
I20240502 08:02:12.743791 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(3598dbda342f49aea2384b5beb17d7a1): 4840 bytes on disk
I20240502 08:02:12.744263 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(3598dbda342f49aea2384b5beb17d7a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240502 08:02:12.787569 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.795250 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.799276 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:12.800797 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.803347 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.821218 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.846629 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.894282 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:12.898479 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:12.900746 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.901772 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.916663 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:12.924266 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:12.937240 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:12.938952 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:02:13.047830 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:02:13.097000 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:13.152398 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.153249 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.154273 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.155779 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.161448 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:13.170509 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240502 08:02:13.175088 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:13.177408 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.181679 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(57f7c358ace742649df52380939ece9d): perf score=0.033392
W20240502 08:02:13.199868 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:13.213882 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.219362 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.038s user 0.010s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":5765,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":559,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":1199}
I20240502 08:02:13.220333 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d): 4586 bytes on disk
I20240502 08:02:13.220824 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20240502 08:02:13.270478 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.274638 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.275003 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.279337 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.295918 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.307206 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.312731 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.366151 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.366541 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.367436 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(3288b438832e488e8257227cb8934565): perf score=0.033360
I20240502 08:02:13.367745 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.370528 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.386165 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.387540 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.389590 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.416920 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.049s user 0.015s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":4613,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":994,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":158,"threads_started":2,"wal-append.queue_time_us":28087}
I20240502 08:02:13.417865 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565): 4840 bytes on disk
I20240502 08:02:13.418414 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 08:02:13.499522 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.499948 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.502305 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.509799 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.511334 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.513837 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.515929 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:13.555189 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:02:13.595702 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:13.598280 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:13.646643 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.656011 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.659178 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.659672 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.663748 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.669181 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.673643 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.690400 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.705936 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.709355 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.714078 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.716322 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.720638 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.729550 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.742532 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling FlushMRSOp(afc9c4d9dd5b4c039686ccd918616859): perf score=0.033381
I20240502 08:02:13.772938 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: FlushMRSOp(afc9c4d9dd5b4c039686ccd918616859) complete. Timing: real 0.030s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":613,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1542,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":595,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":140}
I20240502 08:02:13.774078 28839 maintenance_manager.cc:419] P de423605477a4819a9569821f59f03e4: Scheduling UndoDeltaBlockGCOp(afc9c4d9dd5b4c039686ccd918616859): 4925 bytes on disk
I20240502 08:02:13.774502 28729 maintenance_manager.cc:643] P de423605477a4819a9569821f59f03e4: UndoDeltaBlockGCOp(afc9c4d9dd5b4c039686ccd918616859) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240502 08:02:13.848950 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.849478 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.850101 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.852694 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.853533 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.853717 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:13.867558 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:13.893874 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:13.894609 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling FlushMRSOp(ab785e07fa8248e5889bea8ac755d002): perf score=0.033400
I20240502 08:02:13.909911 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: FlushMRSOp(ab785e07fa8248e5889bea8ac755d002) complete. Timing: real 0.015s user 0.008s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":174,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1682,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":155,"threads_started":2,"wal-append.queue_time_us":158}
I20240502 08:02:13.910764 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(ab785e07fa8248e5889bea8ac755d002): 4838 bytes on disk
I20240502 08:02:13.911777 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(ab785e07fa8248e5889bea8ac755d002) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 08:02:13.970551 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:13.974152 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424): perf score=0.033360
I20240502 08:02:13.976343 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:13.985486 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.985952 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:13.986614 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:13.999552 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.000696 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.026s user 0.016s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":168,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":574,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":163,"threads_started":2,"wal-append.queue_time_us":1371}
I20240502 08:02:14.009263 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424): 4840 bytes on disk
I20240502 08:02:14.009781 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(9c7e7dadff0a4cd08deb043a37a54424) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240502 08:02:14.022011 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:14.071560 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:14.082191 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.084241 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.086143 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.097312 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.099299 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.116714 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.126996 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:14.144832 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:14.171833 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(21d939e5035541a4b003c5d276493d5c): perf score=0.033353
I20240502 08:02:14.182898 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.184810 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.185655 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.186486 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.188028 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:14.189160 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:14.191830 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:14.192121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.214087 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.232522 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.060s user 0.018s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":10236,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1662,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":626,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":154,"threads_started":2,"wal-append.queue_time_us":12874}
I20240502 08:02:14.233428 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c): 4840 bytes on disk
I20240502 08:02:14.233834 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(21d939e5035541a4b003c5d276493d5c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240502 08:02:14.249553 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:14.315198 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.318301 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.323630 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:14.327878 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240502 08:02:14.330804 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.331630 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.332677 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.347070 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:14.423497 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:14.454079 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.459189 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.459240 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.475179 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.475952 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.477305 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.485183 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling FlushMRSOp(57f7c358ace742649df52380939ece9d): perf score=0.033340
I20240502 08:02:14.495725 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.510185 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.516152 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.520328 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.523197 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.528612 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.531891 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.554097 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.574203 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: FlushMRSOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.089s user 0.016s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":31,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":2217,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":608,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":70,"threads_started":1,"wal-append.queue_time_us":220}
I20240502 08:02:14.575148 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d): 4586 bytes on disk
I20240502 08:02:14.575518 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(57f7c358ace742649df52380939ece9d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240502 08:02:14.646179 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:14.651937 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.656816 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.660154 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.665588 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.665835 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.678355 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.690732 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:14.724522 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:02:14.736689 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:14.753276 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(04d358245cdf4b2ebb0b5005f75cf1e0): perf score=0.033385
I20240502 08:02:14.765848 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(04d358245cdf4b2ebb0b5005f75cf1e0) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":188,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1538,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":159,"threads_started":2,"wal-append.queue_time_us":155}
I20240502 08:02:14.766880 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(04d358245cdf4b2ebb0b5005f75cf1e0): 4925 bytes on disk
I20240502 08:02:14.767383 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(04d358245cdf4b2ebb0b5005f75cf1e0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240502 08:02:14.803529 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:14.837678 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.837903 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.840852 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.844774 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.849671 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.851449 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.873934 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:14.984266 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:14.990248 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:14.990295 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:14.994443 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:14.994535 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.003592 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.009807 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.013391 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling FlushMRSOp(6a40382745f84f0eadf9e0ee2e057289): perf score=0.033373
I20240502 08:02:15.041602 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: FlushMRSOp(6a40382745f84f0eadf9e0ee2e057289) complete. Timing: real 0.028s user 0.013s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1428,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":526,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":149,"threads_started":2,"wal-append.queue_time_us":147}
I20240502 08:02:15.042488 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(6a40382745f84f0eadf9e0ee2e057289): 4925 bytes on disk
I20240502 08:02:15.042944 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(6a40382745f84f0eadf9e0ee2e057289) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240502 08:02:15.061342 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(5d1c1fa046b741aa8cf296aa57847fe3): perf score=0.033351
I20240502 08:02:15.075791 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(5d1c1fa046b741aa8cf296aa57847fe3) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1614,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":473,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":173,"threads_started":2,"wal-append.queue_time_us":175}
I20240502 08:02:15.076750 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(5d1c1fa046b741aa8cf296aa57847fe3): 4840 bytes on disk
I20240502 08:02:15.077690 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(5d1c1fa046b741aa8cf296aa57847fe3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
W20240502 08:02:15.131501 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:15.132287 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.136273 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.142436 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.152789 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.153286 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:15.164999 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.176633 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:15.204095 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:15.209076 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:15.216893 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.226099 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.232913 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:15.237330 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.238422 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.246102 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.263114 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:15.286092 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:15.341936 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.342839 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:15.347510 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:15.349237 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.349696 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.350490 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.355983 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.388100 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.480254 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.480798 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.489450 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.493865 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:15.505895 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.507618 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.511171 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:15.561944 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:02:15.593115 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:02:15.601230 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:15.608984 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.611140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:15.611701 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.622278 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.622661 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.634716 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.653353 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.686448 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.693933 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.698603 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.699257 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.705403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:15.719209 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.719522 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:15.742364 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:15.796448 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:02:15.797829 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240502 08:02:15.825402 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.827615 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.833357 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.837961 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.844638 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.855310 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.859848 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:15.960001 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:15.961210 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.964994 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:15.967252 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:15.978027 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:15.981756 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:15.990545 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:15.996389 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.021944 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(167b5f3b818048af81b1350839ebe489): perf score=0.033361
I20240502 08:02:16.037575 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.040537 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.046952 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:16.048063 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.059703 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.067497 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.081552 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(698f494a22d34e659766b97cb24d0293): perf score=0.033386
I20240502 08:02:16.088532 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(167b5f3b818048af81b1350839ebe489) complete. Timing: real 0.066s user 0.019s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":3133,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":2291,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":615,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":137,"threads_started":2,"wal-append.queue_time_us":136}
I20240502 08:02:16.088899 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.089555 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(167b5f3b818048af81b1350839ebe489): 4925 bytes on disk
I20240502 08:02:16.090894 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(167b5f3b818048af81b1350839ebe489) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240502 08:02:16.106775 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(698f494a22d34e659766b97cb24d0293) complete. Timing: real 0.025s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":204,"dirs.run_cpu_time_us":239,"dirs.run_wall_time_us":1709,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":181,"threads_started":2,"wal-append.queue_time_us":172}
I20240502 08:02:16.107894 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(698f494a22d34e659766b97cb24d0293): 4586 bytes on disk
I20240502 08:02:16.108325 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(698f494a22d34e659766b97cb24d0293) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240502 08:02:16.156354 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.156780 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.158288 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.160750 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.161669 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:16.163005 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:16.186539 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.197804 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:16.227571 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:16.295027 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.296063 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.298118 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:16.299641 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.312585 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.321877 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.322013 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.342690 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling FlushMRSOp(a28d406b43194bc6b775707d021618f0): perf score=0.033363
I20240502 08:02:16.362412 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: FlushMRSOp(a28d406b43194bc6b775707d021618f0) complete. Timing: real 0.019s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1543,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":2843}
I20240502 08:02:16.363379 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(a28d406b43194bc6b775707d021618f0): 4925 bytes on disk
I20240502 08:02:16.363924 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(a28d406b43194bc6b775707d021618f0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240502 08:02:16.387694 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:16.439914 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.445199 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.449826 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.460440 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:16.469139 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:02:16.471489 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.483930 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.484066 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:16.487186 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.491986 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.501164 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.507735 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.509949 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.513365 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.527472 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:16.546168 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:02:16.572520 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:16.630043 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:02:16.747997 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:02:16.757377 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:16.839738 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:16.854847 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.862382 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling FlushMRSOp(fddc382ab3164ccdaa86f802c83da8be): perf score=0.033341
I20240502 08:02:16.864740 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.864918 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.866430 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.867900 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.878832 30457 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.879081 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:16.882928 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: FlushMRSOp(fddc382ab3164ccdaa86f802c83da8be) complete. Timing: real 0.020s user 0.017s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":4221,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":659,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":318,"threads_started":2,"wal-append.queue_time_us":363}
I20240502 08:02:16.883841 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(fddc382ab3164ccdaa86f802c83da8be): 4586 bytes on disk
I20240502 08:02:16.884284 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(fddc382ab3164ccdaa86f802c83da8be) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240502 08:02:16.901388 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.902086 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:16.914796 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:16.922349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:16.931466 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:16.932348 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:16.958019 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:17.027541 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:17.052531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.057549 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:17.058701 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.058964 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.060168 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.061584 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.063843 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:17.100059 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:02:17.100823 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240502 08:02:17.162737 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:17.182358 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:17.186946 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:02:17.187685 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.188657 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.188964 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.191407 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:17.191695 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.198254 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:17.272176 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:02:17.301136 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:17.331758 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.341380 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.344616 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.350123 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.359059 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.362452 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:17.366672 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.369148 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.399747 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.402220 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.410832 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.417493 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.419860 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.443603 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:17.475450 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:02:17.512892 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:17.536613 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.538475 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.540164 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.542898 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.543495 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:17.554853 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.563035 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.660313 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.668272 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.670377 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.675807 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:17.675887 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.685164 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.688881 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.789310 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.789597 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.792838 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:17.797272 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:17.802335 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.816397 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:17.820159 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:17.831346 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.861721 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.864156 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:17.876726 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.879532 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:17.886862 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:17.888655 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.893241 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:17.894918 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:17.914664 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:17.997180 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:17.998950 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.000051 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.006036 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:18.011193 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.020527 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:18.021696 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:18.100157 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:18.125838 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:18.126612 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:18.126860 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.141587 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.142599 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:18.143385 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.154026 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:18.212658 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:02:18.229866 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:18.253162 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.263784 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.267009 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:18.282802 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.284854 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:18.301441 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:18.315064 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.325212 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:18.326479 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.330729 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:18.337723 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.339700 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:18.353060 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.363409 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:18.412147 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240502 08:02:18.593907 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:02:18.621449 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:18.670583 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.671447 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:18.679553 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:18.683570 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.688194 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.713604 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:18.717118 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.723699 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.729202 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:18.733420 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:18.741681 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.744871 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.756018 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.756527 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:18.848627 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:18.861271 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:02:18.871129 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:18.875845 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.876087 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:18.882839 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:18.883093 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:18.887708 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.890882 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:18.893630 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.002822 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.003607 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.013190 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.013932 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.014883 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.015740 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.021337 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:19.050796 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:19.113142 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.120213 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.121223 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.128436 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.133761 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.155038 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.161103 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.195952 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.198638 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.204488 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.207284 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:19.211241 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:19.211432 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.212162 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.212200 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:19.224782 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:19.333238 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.334373 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.336279 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.338080 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.344596 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.360126 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.362071 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:19.390285 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:19.422194 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:19.476377 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.484808 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.486004 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.489835 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.490211 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.496054 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.500918 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.524034 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.524201 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.530632 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.542757 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.547828 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.553567 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.560412 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:19.657866 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:19.671612 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.672109 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.674064 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.675434 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.681619 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.691187 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.719396 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:19.731242 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240502 08:02:19.818475 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.821000 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:19.821213 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:19.825176 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.827432 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:19.830796 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:19.834424 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:19.834530 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:19.850353 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.922327 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.925323 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.935623 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:19.937119 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:19.945334 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:19.948277 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:19.949751 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:19.956079 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.013531 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.016525 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.020617 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.024225 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:20.024626 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.030915 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.041121 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:20.089422 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:20.135691 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.136775 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.140074 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.141373 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.149919 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:20.164935 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:20.165599 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.165773 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.267381 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.267832 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.269979 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.271121 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.273813 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.276845 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:20.281901 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:20.307654 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:20.337848 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:02:20.463088 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.465539 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:20.466086 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.467124 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.469002 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.477612 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.491497 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:20.514693 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:20.533037 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:20.608359 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.609289 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.618956 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.624007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.630991 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.631924 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.651607 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.661931 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.666961 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.675406 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:20.686059 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.698155 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.699275 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.699349 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:20.798859 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240502 08:02:20.804901 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.805536 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.807775 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.813483 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:20.818603 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.822705 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.844873 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:20.883201 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:20.931968 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:20.933933 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:20.940847 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:20.943141 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.949118 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:20.950433 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:20.957119 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:20.979820 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:02:21.034727 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:02:21.035773 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:02:21.036744 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:21.074764 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.082003 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.081985 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.089970 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.090428 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:21.104854 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:21.114404 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.123049 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:21.137925 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:21.160820 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.162499 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.162984 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.167901 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.168059 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.184217 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:21.187196 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:21.296090 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:21.301424 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.306212 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:21.306792 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.310920 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.313334 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.333613 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.342126 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.442301 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.451084 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.452546 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.464958 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.466603 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:21.481210 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.495158 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.512820 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.514082 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.519335 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.520131 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.522323 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.523051 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.537871 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:21.549407 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:02:21.622128 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:21.635457 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.637744 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.638448 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.641113 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.647045 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.650076 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.675386 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:21.743484 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:02:21.752216 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:21.769265 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.772118 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.778223 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.784104 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.786041 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.799186 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:21.810238 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:21.822053 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:21.852200 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:21.863569 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:21.863781 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:21.871536 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:21.873998 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.874835 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.876156 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.894956 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.976883 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.979990 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:21.981397 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:21.986328 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:21.996680 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.002207 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.007354 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:22.076400 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240502 08:02:22.097836 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:02:22.126911 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:02:22.131953 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:22.168213 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:02:22.179277 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:22.199278 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.201026 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.202713 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.208140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.218603 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.234078 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:22.234675 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.332885 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.336400 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.338963 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.339109 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:22.346278 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:02:22.349102 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.352746 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:22.371484 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.464368 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.466037 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.467891 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.472313 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.481011 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.482164 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.492635 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:22.501365 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:22.581988 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.582419 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.590785 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.591509 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.592288 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.592473 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:22.613970 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:22.647342 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:22.668584 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.668664 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.669116 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.677505 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:22.683341 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.684721 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.691258 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:22.693737 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:22.805456 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.809386 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.809819 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.812678 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.816700 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:22.824193 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.829694 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.947530 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.954816 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.955770 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.958840 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:22.966534 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:22.969588 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.972743 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.977327 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:22.980568 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:22.990643 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:22.990998 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:22.992483 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.995719 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:22.999439 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:23.025214 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:23.067824 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:23.118721 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:23.136072 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:23.142755 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:23.144057 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.144992 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.145442 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.148136 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.164585 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:23.165982 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.270562 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:23.279562 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.281258 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.289942 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.290508 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.291127 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.302228 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:23.321460 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:02:23.336899 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:23.409756 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:23.417645 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.417686 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.427580 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:23.441766 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.446672 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:23.450513 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.453305 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:23.468854 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:23.470705 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:23.484577 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:23.491670 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.493971 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.494153 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.496045 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.496795 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:23.580897 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:02:23.593416 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:23.621649 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.622484 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.622941 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.630527 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.633873 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:23.644696 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.645996 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:23.711261 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:23.749047 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.750420 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:23.757656 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.759354 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.765173 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.769100 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:23.783318 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.857214 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.860050 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:23.862401 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:23.865135 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:23.875226 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:23.881428 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:23.893646 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:24.082298 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:24.093897 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.095300 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.102211 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.102629 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.108963 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:24.115760 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.122203 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.211746 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.213963 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.221858 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.224870 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.226950 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:24.231462 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.257225 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.301017 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.301853 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:24.302724 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.305485 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.306186 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:24.320436 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:24.330145 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.336632 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:24.345901 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:02:24.353806 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:02:24.415711 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240502 08:02:24.436713 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.437721 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.438603 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.453524 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.469417 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.469566 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.469889 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:24.500180 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:24.580220 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.593353 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.594679 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.595012 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:24.604548 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.618199 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.621181 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.629575 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.631978 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.635645 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.641716 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.651289 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.671662 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:24.679414 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:24.681078 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:24.693159 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240502 08:02:24.698721 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:24.734107 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:24.778945 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:24.779578 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.780822 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.791182 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.796746 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.801652 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.806860 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.911509 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.919515 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:24.923655 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:24.923919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:24.932358 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:24.940698 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:24.972206 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:25.004127 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:25.006639 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.006733 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.016924 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.018219 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:25.027611 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.035370 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.048276 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:25.127501 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.131024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:25.131494 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.133944 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.134491 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.135102 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.151026 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:25.254983 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.255321 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.259346 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.263298 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:25.263330 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:25.263721 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:25.273875 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:25.278339 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.286351 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:25.341600 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:25.369999 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.370682 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.375274 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:25.377254 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.387187 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.391691 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:25.406939 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.473328 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.473820 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:25.476204 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:25.478802 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.488268 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.491636 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:25.492110 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:25.494046 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.606264 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.607770 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.609544 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:25.611725 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:25.620837 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.624598 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:25.626067 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.637677 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:25.649019 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:25.685354 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:25.733155 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.734647 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.734935 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.737664 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:25.746794 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:25.749002 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.751279 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.765344 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:25.911267 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:25.945544 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.946048 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:25.946102 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:25.948740 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.952651 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:25.983675 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:25.984844 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:25.996840 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:26.024315 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:26.057289 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:26.091408 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.092691 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.094460 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:26.094606 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.103190 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.107069 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.114794 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.121732 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.121897 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.137151 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:26.142632 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.144794 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:26.145022 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:26.153100 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.169095 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:26.221508 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:26.282331 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:26.282860 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.283972 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.288658 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.288846 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.298449 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.314553 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.412252 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.418658 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.419840 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.430308 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.433336 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.433902 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.435215 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:26.507506 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:26.535120 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.536041 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.544728 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.547742 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.548673 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.560107 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:26.567564 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:02:26.588689 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.630522 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.635924 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.636893 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.638623 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.652244 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.653668 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.672756 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:26.677196 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:26.724390 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:26.764294 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.766852 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.779126 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.779870 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.780119 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:26.794965 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:26.805475 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:26.812611 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.905392 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.910408 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:26.913137 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.916077 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:26.918923 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240502 08:02:26.921986 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:26.922247 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:26.923267 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.968196 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:26.984344 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:26.989497 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.004005 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:27.004222 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.004480 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:27.008108 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:27.019104 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:27.064148 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:27.087986 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:27.113072 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.113584 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.119764 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.125823 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.126895 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:27.141948 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.145810 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:27.211187 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:02:27.211884 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240502 08:02:27.225755 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:27.264417 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.266238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.269831 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:27.277890 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.284889 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.294631 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.297271 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.352350 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.358152 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.359391 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.362782 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.363548 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:27.372016 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:27.385388 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:02:27.398476 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.467995 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:27.469465 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:27.474973 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.475724 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.478809 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.479351 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.480919 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.481577 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:27.573338 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:27.580029 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:27.616904 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:27.690130 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.690392 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:27.692198 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.694589 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.715472 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.718248 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:27.726457 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:27.727394 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.823993 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.824518 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.825110 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.826074 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.836838 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:27.848079 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.859849 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:27.870143 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:27.953369 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:27.956162 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:27.963465 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.965958 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:27.978586 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:27.985085 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:27.994402 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:27.997076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.072062 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.073762 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.082368 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:28.097052 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.100224 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:28.113750 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:28.117585 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.118131 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.147432 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.150341 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.152879 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.153761 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.161229 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:28.167420 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.173947 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.283012 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.287915 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.291138 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.292308 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.295140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.310559 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.324975 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:28.355688 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:02:28.356442 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:28.383505 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:02:28.401752 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240502 08:02:28.427253 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.431331 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.432045 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.435853 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.440673 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.443535 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.461720 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:28.521696 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.521926 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.527321 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.534870 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.536309 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:28.539738 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:28.544636 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.561304 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:28.569451 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:28.616676 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.619743 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.631635 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:28.633024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:28.640991 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:28.641350 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.644973 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.650558 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.755594 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.757341 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.757992 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.762384 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.763648 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.792555 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.800760 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:28.874565 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.877863 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.881990 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.881990 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.886216 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.891824 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:28.932173 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.953635 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.957476 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:28.966790 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:28.974020 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.979919 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:28.992334 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:28.992339 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:29.016604 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:29.047873 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:29.095897 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:29.099356 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:29.100171 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.104403 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.111388 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.112068 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.135114 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:29.205603 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:02:29.243288 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:29.262686 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:29.263317 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.264941 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.265519 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:29.274840 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:29.285813 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:29.297279 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:29.299156 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.302083 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.304086 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.319586 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:29.325424 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.341436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.348667 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:29.378082 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:02:29.411420 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:29.503772 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:29.556562 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:29.658109 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.659337 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:29.664877 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.677094 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.677070 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:29.677578 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.678815 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:29.734756 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:29.742955 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.745322 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.747344 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.757297 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:29.759447 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:29.770269 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:29.839077 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240502 08:02:29.851819 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:02:29.852288 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:29.872689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.873086 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:29.875854 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:29.894735 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:29.898325 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:29.908036 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:29.910405 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:29.920733 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:30.026559 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:30.027375 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.027487 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.028205 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.038939 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.040355 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.043819 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.050318 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:30.112743 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:30.139387 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.141933 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.146971 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.150828 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.150939 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.162358 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.163415 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.230465 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.234022 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.236101 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.239624 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.249850 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.261160 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.266633 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.369140 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.369490 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.372707 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.375912 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.379586 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.388811 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.396672 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:30.416669 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:30.497422 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.500348 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.501541 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.501917 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.509634 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.513476 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.514185 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.578380 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.582999 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.584546 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.584641 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.585114 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.598831 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.610634 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:30.625312 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240502 08:02:30.684912 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:30.719398 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.724246 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.724298 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.724810 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.733693 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.745096 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.750124 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:30.793452 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:30.879760 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:30.879951 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.882225 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.887570 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.899797 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.900089 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:30.908473 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.909662 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.925998 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:30.926878 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.927292 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:30.937214 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:30.939610 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:30.952131 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:30.956207 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:30.958420 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:02:30.959395 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:02:31.019342 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:02:31.021003 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:31.070237 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:31.070605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.072853 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.074739 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.079676 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:31.082239 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:31.097216 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:31.106660 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.212407 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.212738 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:31.224205 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.224936 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:31.245136 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.248953 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:31.253782 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:31.272841 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.321848 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.323697 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.330052 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:31.335189 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.343808 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:31.347468 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:31.357270 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:31.505756 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:31.573931 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:31.575134 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240502 08:02:31.577316 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.580664 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.583938 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.585083 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.597896 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:31.599053 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:31.610581 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:31.706918 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.709664 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:31.715488 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:31.722458 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.733697 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.752980 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:31.753890 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:31.760628 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:31.773600 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:31.774281 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.776530 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:31.777390 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:31.780050 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:31.791790 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.801661 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:31.807305 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:31.817333 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:31.900497 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:02:31.913105 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:31.920794 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.920902 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:31.923388 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:31.924394 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:31.932441 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:31.932982 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:31.961421 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:32.018179 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:32.056365 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.058296 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.059329 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:32.069528 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.071731 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.083765 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.103044 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:32.148144 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:32.173206 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.179695 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.188845 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.192364 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.194408 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.198531 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:32.199688 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:32.213068 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:32.217065 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:02:32.238008 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:02:32.263358 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240502 08:02:32.267757 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.270937 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.276649 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.278656 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.282384 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.294723 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:32.321879 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:32.334533 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:32.416508 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.420922 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.421190 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.427685 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.438284 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:32.438618 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:32.445185 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.448072 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:32.482599 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:02:32.538540 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:32.543221 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.554975 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.565310 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.572136 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.572659 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.573050 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.641286 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.645426 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.646657 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.646816 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:32.650557 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.652388 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.667837 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.774447 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:32.779299 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.779512 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.782339 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.786028 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.800817 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.809103 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:32.818326 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:32.845963 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:32.931440 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:32.935541 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.938190 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:32.941915 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.942135 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.961884 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.972831 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:32.974153 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.985458 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:32.994374 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:32.995050 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:33.000008 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.007038 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.008441 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:33.113940 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:33.130061 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:33.131776 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:33.132436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.133497 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.134418 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.140831 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:33.156780 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:33.254159 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:33.307521 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:33.330912 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.331815 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.336799 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.337596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:33.337630 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.345228 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:33.379810 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:33.392099 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:02:33.441911 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:33.477030 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.480069 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:33.481524 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.482390 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:33.482542 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.490782 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:33.498291 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:33.533392 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:33.604830 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:33.613111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.615821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.616241 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:33.620759 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.630084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:33.630244 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:33.711519 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:33.740721 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:33.746160 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:33.747946 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:33.752758 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:33.754006 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.755379 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.767292 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.770860 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:33.784346 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.803555 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.810133 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.812687 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:33.816895 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.831082 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.844005 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:33.846143 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:33.857498 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:33.957352 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.957824 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:33.963245 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:33.972538 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:33.973596 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.989158 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:33.998673 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.092343 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:34.094229 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.096609 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.102366 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.103646 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.116106 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.116657 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:34.142807 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:34.186998 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:34.212313 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.212999 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:34.219465 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:34.220172 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.223766 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.225095 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.226050 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:34.264289 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.290661 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.291918 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.296185 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.299868 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:34.304677 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.314508 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.315459 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:34.373224 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:34.395712 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:34.444648 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.450212 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.453974 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.455605 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:34.457463 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.470145 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.470611 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:34.553962 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240502 08:02:34.582732 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.584897 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.585853 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.613010 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:34.613102 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.613360 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.634121 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.657270 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.663267 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.663552 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.670729 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.677868 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.682556 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.682821 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:34.730329 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:34.745682 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240502 08:02:34.768232 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:34.774078 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.779249 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.781385 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.781865 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.785897 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:34.801776 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:34.805663 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:34.818378 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.917470 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.918474 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:34.922540 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:34.930497 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:34.933490 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:34.949466 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:02:34.950173 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:34.955807 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:34.956229 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.036448 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.041353 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.043702 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.060232 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.061146 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.063494 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:35.064893 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:35.190490 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:35.270684 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.273783 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.276047 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.278092 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.279368 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.282913 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.290251 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:35.409629 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.413041 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.415678 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:35.417901 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.434381 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.444952 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.447976 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.491581 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.494256 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.495390 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.506968 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.507881 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.515566 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:35.521906 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:35.531240 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.629758 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.634213 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.638651 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.641886 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.644898 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:35.645412 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:35.646952 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.648389 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:35.735903 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:02:35.737278 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:35.761696 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:35.780684 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.785284 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.787657 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.790819 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.812662 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.821579 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:35.821888 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.856631 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.859165 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.864677 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.869143 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:35.879897 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:35.887096 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:35.892134 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:35.896243 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:35.985790 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:35.988135 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.989925 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:35.993407 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:35.993638 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:36.003178 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.008669 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.119822 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.123016 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.123494 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.129529 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:36.136333 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.150602 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.150923 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:36.160064 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:36.254993 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.257670 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.273833 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.283326 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:36.286662 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:36.292016 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.300283 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.316903 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:36.317898 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:36.333626 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:36.334748 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:36.346280 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.351140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:36.355018 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:36.355577 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.364971 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.374285 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.381006 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:36.480721 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.486029 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.489197 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.489776 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.497442 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:36.499476 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:36.500721 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:36.515362 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:36.531468 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:02:36.601365 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:02:36.623754 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.628633 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.640568 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:36.644779 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:36.644876 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.658427 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.662829 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.714885 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.718827 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:36.719594 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:36.721944 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.730515 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.753340 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.765959 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:36.768379 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:36.776731 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:02:36.811182 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:36.836375 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.838557 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:36.841552 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:36.843730 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:36.848902 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:36.866232 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:36.877564 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:36.884045 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240502 08:02:36.899673 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:37.063730 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.067056 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.069695 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:37.070230 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.070261 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.087275 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.113557 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:37.180923 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:37.184967 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.185047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:37.188594 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.194854 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.199113 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.205240 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.214735 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:37.236332 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:02:37.258016 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:37.293891 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:37.314723 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.323244 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.325726 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:37.328900 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:37.334488 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:37.334959 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.337211 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.358660 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:37.377532 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:02:37.449258 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240502 08:02:37.453711 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.454771 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.459324 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:37.462582 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.483155 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:37.485615 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.507937 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:37.510917 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:02:37.528069 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.531875 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.535575 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:37.543584 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.544371 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:37.560564 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.572872 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:37.663075 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:37.666970 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.676642 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.676636 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.681882 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:37.686731 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:37.692888 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:37.702277 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.713433 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:37.744361 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:37.825366 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.829011 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.829011 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:37.830286 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.844650 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:37.859561 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.864038 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:37.871639 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:37.909047 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:37.914546 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.916898 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:37.919654 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:37.934036 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:37.936702 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:37.959775 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:37.989735 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:38.031515 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.036242 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.037992 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:38.039633 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.039973 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.048738 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:38.071130 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:38.168836 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:38.171547 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:38.174192 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.177140 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.177569 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.180012 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.181355 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.196658 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:38.248530 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:38.317116 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.320149 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.320293 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.336119 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.337977 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:38.340219 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:38.350975 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.354230 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.363116 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.375247 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:38.376106 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:38.377365 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.384703 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.391865 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:38.402220 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:38.453804 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:38.511492 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.515439 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.518117 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.518952 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:38.522374 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.522743 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:38.547329 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.644671 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:38.645927 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.647472 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.648873 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.653844 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:38.656648 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.667651 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:38.744701 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:02:38.755424 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:38.761907 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.764633 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.764933 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.765035 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:38.774012 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.793654 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:38.800314 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:38.809068 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:38.983978 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:38.983978 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:38.989759 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:39.005679 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.006659 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.006713 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.012483 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:39.084115 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:39.116956 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.119971 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.120364 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.120754 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.150079 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.154310 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:39.160610 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:39.165212 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:39.175870 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:39.187914 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.190465 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:39.196803 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:02:39.199263 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:39.199975 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.205543 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.209635 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:39.212678 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.234431 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.347600 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.349835 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.353880 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:39.359555 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.360949 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.361979 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.376370 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:39.458390 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:39.487135 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.487890 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.493090 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.500962 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.505653 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.510632 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.532115 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:39.542726 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:39.571659 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.574275 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.582885 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.587700 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:39.588666 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.599138 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.601649 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:39.620137 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240502 08:02:39.639550 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:39.692371 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.698257 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.706053 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.707614 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.709048 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:39.716162 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.744235 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:39.765625 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:39.835994 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.840562 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:39.844177 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.847234 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.847226 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.861488 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.887867 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.956678 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:39.957921 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:39.964738 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:39.970346 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:39.972592 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.001711 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:40.008250 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:40.019330 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240502 08:02:40.045013 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.046633 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:40.048851 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:40.052888 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.053561 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.062860 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.075803 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:40.078617 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:40.084759 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.184331 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.186769 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.191488 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:40.198094 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:40.198305 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:40.203356 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.216109 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.216914 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:40.240955 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:40.349596 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.349747 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.358882 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:40.369802 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.374451 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:40.379483 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.392987 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:40.393307 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.402344 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.405022 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.407505 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:40.410862 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:40.424499 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:40.424667 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.440920 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:40.450763 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:40.545874 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.546970 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:40.547255 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.555423 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.556416 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:40.568987 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.580116 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:40.620011 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:40.676662 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:02:40.688944 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:40.746941 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:40.750897 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.753968 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.756891 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.758411 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:40.770581 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.787534 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.889572 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:40.894358 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:40.894994 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.897204 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:40.899068 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:40.901875 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:40.909613 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.012689 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.013554 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.014088 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.020361 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.040519 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.042872 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.051364 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:41.088970 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:41.130878 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.139528 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.142318 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.143649 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.149160 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.155030 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.173588 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:41.211859 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:41.229652 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.234686 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.234864 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.236498 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.239537 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:41.260942 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:41.279258 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.290279 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:41.349028 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:41.388864 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.390080 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.393646 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.403705 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.405292 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.415359 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.418257 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.522163 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.524812 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.535202 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.539629 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.549693 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.550194 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.552971 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:41.575199 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:41.585469 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.588982 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:41.600862 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:41.603751 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.607501 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.609303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.613337 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.613329 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:41.629457 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:41.630110 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:02:41.668066 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:02:41.686770 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240502 08:02:41.733655 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.734669 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.735388 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.740298 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.741147 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.754742 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.761168 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:41.804370 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:41.824329 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:41.868845 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.872615 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:41.876776 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.881127 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.886091 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.889269 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.911453 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.936753 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:41.941838 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:41.942456 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.947438 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:41.957616 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:41.967454 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:41.983727 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:41.993227 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:42.013810 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:42.073356 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:42.075529 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.078528 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.079129 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:42.084647 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:42.086119 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:42.096630 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.096619 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.206660 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.207070 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:42.211233 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.212317 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:42.223377 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.225275 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.253912 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.333597 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:42.346477 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.351006 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.351606 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.352702 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.359586 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:42.367066 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:42.375157 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:42.427713 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.432389 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.433373 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:42.440757 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:02:42.444931 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:42.451819 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.461978 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:42.476646 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:42.488004 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:02:42.540661 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:02:42.580387 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:02:42.600873 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:42.692886 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.693894 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.694053 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.718005 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.719776 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:42.721402 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.722980 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:42.789539 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:42.793519 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.800902 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.803480 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.803529 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:42.805742 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.827240 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.827374 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:42.853042 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240502 08:02:42.859195 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:42.931679 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:42.934260 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.941699 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.942005 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:42.942855 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:42.944336 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:42.962395 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:42.984447 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:43.014253 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240502 08:02:43.029484 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:43.070176 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.081439 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.088726 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.095152 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.099393 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.108700 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.121549 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.142036 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.144115 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.160925 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.163300 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.169346 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.176486 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.185420 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:43.219087 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:43.290940 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.290956 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.292258 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.300199 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.300616 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.300940 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.318220 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:43.376025 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:43.433972 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.434064 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.434962 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.445048 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.450134 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.455848 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.466480 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.516250 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.520680 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.522495 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.525040 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.537336 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.538486 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.543514 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:43.551543 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:43.634624 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.636643 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.640964 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.648854 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.655138 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.655983 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.659927 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.789414 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.790788 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.791591 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.792562 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.804920 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.808135 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.843351 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:43.866103 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:43.918200 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.919725 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:43.920171 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:43.927385 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:43.936363 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:43.937127 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:43.971453 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:44.008625 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:44.012574 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.014001 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.014314 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:44.014860 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.024112 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:44.035645 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:44.044998 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.052258 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:44.063952 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:02:44.148171 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:44.150386 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:44.153883 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:44.161031 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.164119 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.164721 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.165235 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:44.169092 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:44.172495 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:44.180555 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:44.272845 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:44.290212 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:44.292518 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.297950 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:44.309578 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:44.311698 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:44.312861 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:44.327826 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.334023 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:44.344264 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:44.344549 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.372085 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.378443 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.380185 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.386360 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:44.386648 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:44.570405 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240502 08:02:44.578358 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:44.652745 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:44.704905 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.716061 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.716331 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.718691 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:44.731642 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:44.733839 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:44.750017 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.752197 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.759711 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:44.768824 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:44.774971 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.780988 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.793093 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:44.805320 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:44.866482 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:44.914526 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.914860 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:44.918615 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:44.920542 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:44.922627 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:44.933868 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:44.935504 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:44.955791 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:45.059150 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:45.060127 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.070290 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.072342 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:45.073702 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.073971 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.079628 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.086383 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:45.176175 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:45.183104 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.187295 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:45.187367 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.188304 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.189532 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:45.194213 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.225812 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.251410 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.254971 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.260320 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.272737 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.284080 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:45.292349 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.303974 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:45.403908 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.407752 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.408350 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.419968 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.420091 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:45.425865 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:45.429466 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:45.439038 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:45.461426 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:45.541404 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.543659 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:45.543766 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:45.549003 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.563165 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:45.563503 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.563627 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.570540 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:45.593179 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:45.633677 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.637061 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:45.637331 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:45.638286 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:45.644999 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.648008 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.664765 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.676362 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:45.705268 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240502 08:02:45.750198 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.750996 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:45.755774 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.759361 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:45.759857 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.765311 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.793031 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.885612 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:45.888981 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:45.890208 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.890233 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:45.890585 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.899363 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:45.906548 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:45.932698 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:46.021731 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.029605 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.029610 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.042001 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.043067 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.046200 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:46.067246 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:46.090065 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:46.097971 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.099574 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.113907 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.118464 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.123029 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:46.125324 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.132388 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:46.181311 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:46.234438 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.235975 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.243871 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:46.256392 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.257333 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.265187 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.268662 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.387635 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.389125 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:46.395995 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.397933 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.401716 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.401831 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.429291 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:46.483387 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:02:46.528095 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:46.572906 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.578433 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:46.578943 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:46.583460 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:46.584950 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:46.587594 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.589869 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.592526 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.597546 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:46.617452 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:02:46.670472 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:02:46.711484 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.711874 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.733193 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.736518 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.736826 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.736836 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:46.744961 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:46.765223 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:46.797276 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.811607 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.818140 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.819430 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.822860 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:46.825121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.831163 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.927317 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.928766 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:46.931962 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:46.933167 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:46.934939 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:02:46.936583 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:46.938030 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:46.971450 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:46.996599 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:02:47.031358 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:47.038316 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:47.069916 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.070614 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.072921 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:47.073197 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:47.074831 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.088541 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.090162 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.090658 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:47.162287 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:47.201201 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.218974 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.220222 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.222901 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.230849 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.231676 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.243804 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.289922 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.290388 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.291285 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.296749 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.299890 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.308208 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.309908 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.426856 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.427073 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.433854 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.435729 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.436214 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.436713 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.437523 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:47.463429 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:47.482702 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:47.563984 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.564494 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:47.573583 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:02:47.573761 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.581426 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.582201 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.584753 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.613647 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:47.631603 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 37 similar messages]
I20240502 08:02:47.673203 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.677028 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.689625 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.690354 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.691095 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.716544 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.721873 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:47.743093 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:47.763604 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.765358 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.770620 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.771235 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.783911 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.784621 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.793349 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:47.852264 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:02:47.878495 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:02:47.879338 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:02:47.888751 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:47.893918 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:47.905126 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.905126 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:47.907590 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:47.910415 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:47.920961 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:47.933532 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:47.940893 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:48.008445 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240502 08:02:48.054545 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.055382 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.061470 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.066684 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.083748 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.083846 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.092156 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.098332 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.114272 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.121744 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.127820 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.134166 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.135682 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:48.141111 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:48.152627 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:48.197677 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:48.397154 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:48.423846 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.428162 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.435577 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.442981 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.444434 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.444535 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.463970 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.519042 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.520907 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.522600 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.525491 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.532727 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.537899 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.547616 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.619611 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.623296 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:48.623782 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:48.625440 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.629690 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.630211 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.636680 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.640292 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:48.664952 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240502 08:02:48.715368 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:48.752671 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.756716 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.756979 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.762931 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.763672 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.768049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.783143 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.893850 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.898682 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.907756 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.908511 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.910470 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:48.931712 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:48.933210 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.968787 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.970106 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:48.974288 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:48.974442 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.977752 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:48.985059 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:48.988672 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:48.991696 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:49.034376 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:49.113137 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.113857 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.121723 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.123045 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.128379 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.134699 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.145443 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:49.191758 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:49.205570 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:49.262245 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.263597 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.277597 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.288200 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:49.290136 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:49.302008 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:49.305850 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.322332 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.364782 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.366508 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.371017 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.375732 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.378929 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:49.398496 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.411186 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:49.484177 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:49.490854 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.490854 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:49.492429 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.499822 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.507447 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.511013 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:49.520707 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:49.524365 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.612084 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.615049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.619916 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.636121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.637616 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.640167 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.641050 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:49.646641 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:49.683562 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:49.745627 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.747977 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.748994 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.761788 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.762959 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:49.764676 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.772534 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:49.794873 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:49.839042 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.842571 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.843468 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:49.849424 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.856739 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.861366 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:49.871057 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:49.896991 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:49.984612 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:49.985839 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:49.992594 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:49.993148 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:49.993206 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.009263 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.020884 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:50.049387 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:50.102306 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.108968 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.109798 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.110702 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.111075 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.117038 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.123462 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:50.191422 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:50.263036 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:50.325906 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.326563 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.329020 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.340792 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.343705 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.354064 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.360692 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:50.468705 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240502 08:02:50.476284 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.485903 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.486662 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.486826 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.495123 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.496403 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.515813 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.517464 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.517720 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.520382 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:50.520804 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:50.537576 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.545917 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.549180 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.566098 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:50.588007 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:02:50.588132 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:50.682356 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.687484 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.687847 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.688736 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.689203 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.696652 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.722890 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:50.745929 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:02:50.790831 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:50.837210 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.841480 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.843072 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.846002 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.848134 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.853138 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.866164 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:50.889618 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:50.932956 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.935047 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:50.939769 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:50.940349 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:50.940882 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:50.956650 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:50.980666 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.051065 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.052920 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.057579 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.059891 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.062120 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.063006 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.083654 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.197328 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.199620 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.199929 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.216607 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.216957 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.217236 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.222651 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:51.236342 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:02:51.279517 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:51.284912 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:51.340118 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:51.376045 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.379300 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.381362 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.390372 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.390970 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.403573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.415573 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.453326 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.468752 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.469426 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.470444 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.483690 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.483978 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.503088 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.591668 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.591964 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.594987 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.600512 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.601809 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.609429 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.620755 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:51.666271 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:02:51.672803 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:02:51.673557 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:51.683773 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:02:51.690603 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:51.760687 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.761235 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.763388 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.763407 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.768702 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.772019 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.782348 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.788159 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.810698 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.815546 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.820907 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:51.822852 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.830569 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:51.857322 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:51.862179 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.969386 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.969981 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:51.978499 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:51.980917 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:51.982851 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:51.983131 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:52.011121 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:52.112010 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:02:52.164085 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:52.169049 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:52.170418 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.177750 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.181092 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.181146 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.182118 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:52.188658 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.189148 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:52.216208 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240502 08:02:52.295059 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:02:52.309572 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.310726 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.315496 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.315596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.318704 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:52.333400 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.342856 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.449684 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.456615 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:52.456624 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240502 08:02:52.461153 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.465082 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.478425 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.484393 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:52.495275 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:52.495561 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:52.574872 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.578155 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.580163 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.581306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.593927 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.615121 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.615303 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:52.642683 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:02:52.644121 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:52.670742 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.677433 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.683714 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.683831 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.684227 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.693111 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:52.695384 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:02:52.701519 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:52.703531 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:52.759430 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:52.819845 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.820295 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.829458 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.832700 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.834054 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.839259 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:52.863298 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.958477 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:52.960621 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:52.962641 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:52.964218 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:52.967027 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:52.974741 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240502 08:02:52.980393 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:52.985810 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.004745 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.055619 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.059981 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.061604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.066110 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.071734 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.072610 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.088923 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:53.094761 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:02:53.146479 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:02:53.163157 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:53.198967 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.201579 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.207521 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.213397 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:53.215113 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:53.221460 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:53.221524 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.223275 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:53.265714 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:53.343533 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.345870 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.354969 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:53.369256 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.376281 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.383376 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.385358 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.397133 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.403990 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.421546 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.425113 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.428735 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.429788 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.437433 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:53.556627 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.559353 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.564574 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:53.566226 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.576411 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.576748 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.581343 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:53.630031 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:02:53.645511 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:53.703685 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.704236 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.706739 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.719573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.729188 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:53.730751 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:53.730978 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.748744 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.816989 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.821061 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.822912 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.826830 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.842120 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.851701 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.864132 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:53.876237 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:53.922256 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.922724 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:53.924196 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:53.929194 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:53.932778 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:53.934866 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:53.948338 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:54.113723 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:54.167865 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.171031 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:54.177531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.180783 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:54.183348 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.186602 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.196349 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:54.273042 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:54.288897 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:54.292629 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.293104 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:54.296531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.296921 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.302485 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:54.311762 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.336148 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:54.346305 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:54.408594 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:54.435122 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.435415 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:54.440752 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.442919 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:54.444303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.444319 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.449970 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.561807 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.565861 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:54.578907 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.583369 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.606395 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:54.615720 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.618700 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.630816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.633685 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:54.638984 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.642621 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.651213 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.658922 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.667178 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:54.692764 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:02:54.702373 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:54.729279 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:54.787385 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.792665 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.796109 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:54.799255 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.811012 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.816713 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:54.830478 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:54.839857 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:54.840199 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:54.942924 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:54.942982 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:54.944284 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.959138 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:54.969470 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:54.983522 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:54.988962 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.034089 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.043463 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.043685 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.044570 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.055471 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:55.063957 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.065019 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:55.117415 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:55.139110 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.141965 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:55.144912 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:55.145056 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.145717 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.148705 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:55.155498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.162539 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:55.246819 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240502 08:02:55.297291 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.297322 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.298599 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.301872 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.307984 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:55.333097 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.336217 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:55.374612 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:02:55.385902 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:55.425122 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.429698 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.434917 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.438288 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.439036 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.448717 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.451052 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:55.512118 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.515130 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.518754 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.523061 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:55.530959 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.533159 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.545728 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:55.589804 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:55.598603 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:55.644887 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.647078 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.652714 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.655906 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:55.662432 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.664986 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.674132 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.796054 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.796867 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.796902 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.812669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.816936 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:55.822244 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:55.827749 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:55.836388 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:55.851267 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:55.854243 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.860519 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:55.867836 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:55.869356 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:55.873790 28639 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.873790 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:55.876355 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:55.909036 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:55.941753 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:56.055370 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:02:56.074338 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:56.208030 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.212124 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.223286 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:56.224519 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:56.229449 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.232647 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.252182 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.262591 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.267591 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.273697 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.275424 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:56.279320 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:56.301870 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.314584 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.419962 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.421517 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.421630 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.423673 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.431294 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.445263 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:56.448388 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:56.504985 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:56.506551 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:02:56.538439 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:56.559207 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.560528 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:56.563324 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.569128 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.572677 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.576709 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:56.577127 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.671604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.673735 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.675473 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.677909 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:56.698004 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.721192 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.725490 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:56.770267 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:56.770823 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:02:56.771850 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.771932 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.772065 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:56.777734 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:56.785732 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:56.802981 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.803326 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:56.916270 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:56.919096 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:56.920326 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:56.922284 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:56.925827 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:56.940069 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:56.962255 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:56.963865 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:56.986924 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.087754 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:57.088291 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.090976 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.091612 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.099952 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:57.103276 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:57.106204 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.109426 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.166507 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.170359 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:57.173888 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.180629 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.182422 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.190061 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.211764 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:57.258716 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:57.314358 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.316321 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.328130 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.329099 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.330933 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.331429 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:57.341078 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:57.343617 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:57.370174 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:02:57.460428 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:57.460954 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.472376 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.487565 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.498759 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.500569 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:02:57.511345 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:57.515806 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.516893 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.517006 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.527436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.528609 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.530098 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.541854 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.559221 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:57.652915 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:57.657884 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:02:57.661720 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:02:57.670172 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:57.671283 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.674793 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.684659 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.688277 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.691833 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.711069 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:57.734761 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:02:57.744024 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240502 08:02:57.776846 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240502 08:02:57.817124 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:57.832392 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.840926 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:57.844156 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.857579 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.860396 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.887365 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.901482 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:57.906033 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.909185 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:57.919967 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:57.927851 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:57.931356 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:02:57.939002 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:57.941576 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:57.976727 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:58.206734 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:02:58.206892 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:02:58.270093 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.279850 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.282548 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.287456 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.288440 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:58.288784 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:58.303927 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:58.332595 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.348088 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.349807 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.351023 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.359594 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:58.375269 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.375945 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.379882 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:58.414939 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:02:58.449143 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:02:58.482662 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.488087 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.489341 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.490641 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:02:58.491760 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:58.500093 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.500862 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:58.508702 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:02:58.513386 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:58.520736 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:02:58.612977 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.616237 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.628731 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.629217 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.630610 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.648738 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.673067 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:58.723559 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.724633 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.731904 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.732198 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.740093 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.764272 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.766494 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:02:58.794628 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:58.818656 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.829890 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.836669 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.840844 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.842051 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:58.843814 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:58.844100 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:58.935415 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:02:58.954567 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:58.963053 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.964102 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:58.970165 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:02:58.971778 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:58.974085 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:58.975826 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:58.978984 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:59.029289 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:59.105163 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.105546 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.112639 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.116853 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.117000 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.135316 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.140949 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.150563 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.153292 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.160643 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.172847 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.184343 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.187713 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.188015 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:59.228374 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:59.306160 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.309062 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.315006 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.315186 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.316563 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.335755 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.351013 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:59.437567 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:02:59.444537 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.450825 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.453616 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.453642 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.456072 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:59.461557 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:59.471374 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.473536 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.568608 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.570436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.586436 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.597802 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.598536 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.617950 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.631500 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.645743 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.648304 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.657424 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.659024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.660015 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.662302 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.696342 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:59.716252 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:02:59.785238 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.787925 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.788763 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.795564 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.798017 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.808679 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.808987 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:02:59.827112 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:02:59.872349 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:02:59.940111 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:02:59.943156 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.945916 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:02:59.962989 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:02:59.966475 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:02:59.967224 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:02:59.982561 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:02:59.993019 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:00.130370 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:00.130929 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.138000 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.140806 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.146694 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.153429 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:00.158638 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:00.189508 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:00.270961 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240502 08:03:00.289173 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.301908 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.303313 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.306334 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.313791 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:00.316257 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:00.327144 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.351722 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.357527 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:00.365285 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.380983 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:00.381734 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.385730 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.392535 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:00.399786 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:03:00.492030 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:00.493780 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.497581 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.499610 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.509382 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.511896 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.516132 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:00.518327 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:00.618953 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:00.630383 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:00.636420 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240502 08:03:00.638186 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:03:00.639499 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.639822 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.646024 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.647619 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:00.649660 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.674430 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.758200 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:00.758956 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:03:00.761250 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.770186 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.772204 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:00.772925 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.773732 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.792354 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.841084 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.845093 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:00.854148 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.854635 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.856442 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:00.873497 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.893088 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:00.901827 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:00.943467 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:03:00.947554 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:00.989940 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:00.992487 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:00.996631 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.997213 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:00.998855 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.009773 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:01.018254 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:03:01.019219 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:01.058718 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:01.129382 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.132436 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.136124 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:01.142552 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.148689 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:01.155357 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.164232 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:01.189953 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:01.233379 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.235378 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.239830 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.241003 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.242511 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:01.249141 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.257498 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:01.305368 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:01.333076 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.334332 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.342864 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:01.344029 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.349689 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.350140 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:01.352866 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:01.416504 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:03:01.424381 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:01.499636 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.500634 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.504374 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:01.508970 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:01.510244 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.514269 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.531633 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:01.539376 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:01.632087 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.640038 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.640655 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.647001 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.652148 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:01.654564 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:01.665969 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.719748 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.723917 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.729350 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.733472 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:01.739686 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.749603 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:01.751952 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:01.752496 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:01.875859 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:03:01.924805 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:03:01.979766 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:01.981303 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:01.990785 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.992108 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:01.993656 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:01.995760 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.019562 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.105587 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:02.107138 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.110334 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.112210 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.117202 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.117507 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:02.119685 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:02.144750 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:02.172446 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:03:02.207374 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:02.247682 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.255299 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.259399 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.260985 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:02.267678 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.268518 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.279934 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:02.328835 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:02.385480 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.388152 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.389101 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.390787 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:02.396359 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.400180 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.405670 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.469589 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.472630 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.476305 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.476933 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.481572 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.482394 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.486693 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:02.531786 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:02.607908 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.608099 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.610826 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:02.613255 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.614174 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.622680 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:02.632346 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:02.633706 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:02.649206 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:02.703433 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240502 08:03:02.753830 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.754271 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.755331 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.769053 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.769104 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:02.777431 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.788945 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:02.819020 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:03:02.848379 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:03:02.855234 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:03:02.886132 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.887212 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.888031 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:02.889050 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.895429 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.901995 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.930722 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.967214 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:02.971508 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.975487 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:02.976792 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:02.979079 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.979789 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:02.999598 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:03.033953 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:03.078989 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240502 08:03:03.080958 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:03.106712 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.108764 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:03.114111 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.118017 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.118393 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.119709 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.147387 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:03.257180 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:03.259927 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.262392 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:03.275280 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.281100 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:03.282244 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.287196 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.305044 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.328904 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.332831 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.333133 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:03.336436 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:03.337536 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.341570 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.364921 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:03.365198 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:03.463474 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:03.474570 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:03.475719 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.476613 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.485066 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.485641 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.493929 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.502882 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:03.556322 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:03:03.600195 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:03.604790 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.606654 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.609053 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:03.612963 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.621161 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.623142 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:03.633491 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:03.669740 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:03.734493 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.735782 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.737352 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:03.739338 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.740404 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.742049 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:03.752806 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:03.764716 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:03.796159 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:03:03.798898 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:03.896597 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:03.950345 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.952795 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:03.953720 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:03.954386 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:03.959795 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:03.962261 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:03.986227 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:03.987461 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.096204 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.097712 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:04.100495 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:04.103650 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.114007 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.127326 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.136507 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.139830 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.156131 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.157898 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.188169 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.188647 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.190133 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.197049 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.202196 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:04.301302 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:04.317116 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.319331 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.319679 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.331373 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.332628 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.335399 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.348846 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:04.388891 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:03:04.389891 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:04.474059 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:04.474570 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.483757 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.492660 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.504632 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.506395 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.510187 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.512060 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.513517 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.518675 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.520205 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.537226 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.557547 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.565349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.567023 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.679601 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.681066 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.686709 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.689218 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.704957 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.709534 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.725231 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.842375 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.847520 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.854574 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:04.856331 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.857143 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.863662 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.865566 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:04.879859 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:04.880443 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:04.881100 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:04.948557 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:04.953344 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.956669 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.960552 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:04.972537 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:04.979995 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:04.989785 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:05.059295 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.060175 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.067915 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.068365 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.072930 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.073666 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:05.088840 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:05.135949 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:05.197963 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.201208 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.201372 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.206305 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:05.217772 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:05.218405 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.225682 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:05.247100 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240502 08:03:05.253265 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:05.260396 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:05.346160 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.351676 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.351888 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:05.353528 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.356772 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:05.371665 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:05.382048 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.393328 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.425197 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.432307 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.433450 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.440115 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:05.449851 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.450654 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:05.458195 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.570410 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.575644 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.576586 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.583065 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:05.591809 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.592626 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:05.593644 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240502 08:03:05.595060 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:05.706799 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:05.707463 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.707525 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.721603 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.722378 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:05.735263 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.736935 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:05.764165 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:05.797433 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:03:05.856612 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:05.911736 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:05.914060 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.918000 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:05.919353 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:05.922752 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.929597 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:05.945477 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:05.986860 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:06.058584 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.071488 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.071794 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:06.075726 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.075737 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.077030 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.089008 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.091976 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:06.105118 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:06.112886 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:06.127990 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.139183 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.141335 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:06.165454 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.166544 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:06.171871 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:06.277932 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.282531 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.283924 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.291944 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.292435 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:06.298635 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:06.315035 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:03:06.315304 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:06.319252 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:06.388164 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:06.431901 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.436097 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:06.436745 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.445390 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.453578 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.464478 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:06.465139 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:06.483795 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:06.503481 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.527530 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.532979 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:06.534373 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.535064 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.549842 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:06.565047 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.645604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.645715 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.653280 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.655508 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:06.656054 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.656980 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.658964 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:06.670389 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:06.785449 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.788566 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:06.793601 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:06.795523 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:06.795831 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.806274 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.810276 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.817386 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:06.818037 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:06.888895 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:06.935129 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:06.943784 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.943955 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:06.954645 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:06.955106 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:06.955407 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:06.979391 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:06.983943 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:03:07.005726 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.006001 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.007807 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.009482 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:07.012753 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.015523 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.036355 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:07.045414 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:03:07.115469 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:07.136108 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.136966 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.137857 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.139067 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.140980 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:07.157929 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.175483 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:07.214388 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:07.298023 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.299577 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:07.301839 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.303687 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.315819 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.329005 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.353057 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.353310 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.354806 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.364169 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.368950 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.372872 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:07.373296 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.375921 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:07.381485 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:07.500077 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.501682 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.502652 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.504917 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:07.523428 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.526101 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.532364 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:07.653550 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:07.658751 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:07.709147 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.710042 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.719601 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.726603 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.727344 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.727563 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.736960 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:07.802974 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240502 08:03:07.845119 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.845621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:07.849452 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:07.851639 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:07.855007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:07.861052 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:07.879642 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:03:07.882483 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:07.910097 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:03:07.910567 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:03:07.912000 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240502 08:03:07.937631 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:07.992765 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:07.994488 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:07.998345 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.004746 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.006408 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.009687 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.011020 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.033862 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.112026 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.116303 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.121856 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.124406 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.133160 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.133935 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.142129 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.200438 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.201007 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.207103 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.210112 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.212147 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.213698 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.218644 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:08.252974 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:08.341984 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.344105 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.349766 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.351454 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.352149 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.358265 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.379626 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:08.398553 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:08.497452 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.498198 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.501835 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.503561 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.518939 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.521461 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:08.536989 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:08.538419 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.553368 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.558403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.566279 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.578291 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.588294 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.589848 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:08.602705 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:08.613905 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:08.635521 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:03:08.678709 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:08.716673 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.721032 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.722862 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.723127 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.727072 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.737710 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.763494 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:08.775624 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:03:08.838089 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:08.874629 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.876403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.880667 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:08.887049 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:08.891810 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.894207 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.897625 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.920683 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.947011 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:08.947353 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:08.950583 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:08.953454 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.955756 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:08.962260 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:08.971050 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:09.074788 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:09.080127 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.082639 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.085348 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.097067 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.097399 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:09.097860 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:09.123721 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:09.131285 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:09.223109 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:09.227169 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.230890 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.230947 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.231737 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.239053 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:09.252897 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:09.261214 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:09.342204 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:03:09.356791 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:09.359756 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.363651 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.368428 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.372205 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.373198 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.375164 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:09.376663 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:09.391492 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:09.448812 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:09.454836 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.458815 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:09.459964 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.461915 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.467166 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.496613 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:09.548219 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:03:09.673120 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:09.707603 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.710786 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.716342 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.719137 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:09.721814 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.733739 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.745272 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:09.755920 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:09.842126 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.845476 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:09.849005 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.849400 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:09.860956 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.874651 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:09.880091 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.982700 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:09.985888 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:09.996876 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:09.998833 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.000487 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.003509 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.004268 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:10.050277 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:10.072762 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:10.114894 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.115175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:10.125818 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.127256 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.138010 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.146087 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:10.155905 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:10.183467 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:10.187179 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.188666 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.193328 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:10.195359 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:10.204566 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.205462 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.212978 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:10.265111 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240502 08:03:10.309072 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:10.347826 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.351608 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.354429 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.360930 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:10.362573 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.386303 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:10.388368 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:10.439579 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240502 08:03:10.492609 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.501220 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.508147 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.509711 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:10.510087 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.511739 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.525012 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:10.549793 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:10.571938 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:10.599169 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.603073 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:10.607846 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.612406 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.623427 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:10.637801 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.643394 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.707571 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.708804 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.710662 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.713517 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.715939 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:10.719195 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:10.729712 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:10.731902 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.850397 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.854210 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:10.855299 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.857304 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.857412 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:10.879626 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:10.880394 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:10.971889 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:10.981299 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:10.981956 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.000175 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.000548 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.010044 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.014716 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:11.016358 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:11.033376 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:03:11.058593 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:11.065182 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.065853 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:11.067510 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.079252 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.082093 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.089751 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.091637 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:11.177028 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:11.182467 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:11.212855 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.213059 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:11.213698 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.218641 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.226691 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.242188 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:11.249075 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:11.280355 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:11.368041 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:11.377882 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.383435 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.384835 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.396281 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:11.396522 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.411321 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.412096 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.419230 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.421236 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.436897 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:11.440387 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:11.448698 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.456871 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:11.487329 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:11.533735 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:03:11.548780 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:11.645356 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:03:11.698637 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:11.787523 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.787652 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.791236 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:11.797632 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.811089 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.826050 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.826331 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:11.858784 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:11.860972 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:11.864938 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.868072 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:11.872880 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:11.873020 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:11.882005 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:11.900029 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.009943 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.010182 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:12.011648 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:03:12.012727 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:12.015439 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.018702 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.031306 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.044684 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.049217 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:12.055346 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:12.153550 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.163563 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.164251 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.171419 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.176707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.196427 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.215351 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:12.229542 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:12.280419 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.286032 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.296116 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.300321 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:12.302323 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:12.309451 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.322391 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.324206 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.392172 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.397717 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.400322 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.403337 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.405321 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.409803 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.433526 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:12.476230 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:12.491092 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:12.537405 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.540302 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.543754 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.553005 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.553288 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.553809 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.581609 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.676062 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.682009 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.688057 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.688179 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.693065 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.716650 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.726910 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.764216 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.767647 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.770723 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.780709 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.782981 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:12.788781 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.801096 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.916249 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.918231 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:12.918560 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:12.922362 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:12.928462 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:12.936687 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:12.938289 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:03:12.940404 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240502 08:03:12.959663 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:12.996793 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:03:12.998273 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240502 08:03:13.024026 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:13.058265 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:13.060905 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:13.069831 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:13.076318 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.077729 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.084401 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:13.092543 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:13.103732 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:13.115818 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:13.119216 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:03:13.132331 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:13.139040 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.139468 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:13.148283 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:13.151378 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.178867 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.183154 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.296691 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.297147 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:13.304315 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.312011 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:13.320634 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:13.322434 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.323647 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.324062 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:13.466238 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:13.466715 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.473680 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:13.476310 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.483654 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.489449 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:13.493312 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:13.512643 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:13.513211 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:13.558328 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:13.560896 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:13.561820 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.562034 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.563740 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:13.564800 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.566983 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.569118 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:13.660410 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240502 08:03:13.718763 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:03:13.820963 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:13.829500 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.836387 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:13.840766 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.842984 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:13.857380 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.859109 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.869313 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:13.945991 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:13.947072 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.947962 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:13.970263 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.976246 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:13.990197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:13.990520 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:14.000180 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:14.008390 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:14.050343 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.056073 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.058390 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.065927 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.069041 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.074667 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:14.092756 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:03:14.094857 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:14.097105 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.200794 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.203639 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.205096 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.206990 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.207271 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.222682 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.228679 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:14.323168 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:14.336661 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.341667 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.355544 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.366282 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.371474 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.379554 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:14.380367 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:14.381870 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:14.435920 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.437959 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.438239 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.438930 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.439786 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.442037 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.463675 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:14.490037 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:14.568847 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.570607 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.575374 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:14.585498 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:14.586000 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:14.586509 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.594287 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.611925 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:14.615578 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:14.717393 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.721271 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.721498 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.739187 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.742646 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:14.742755 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.749505 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.796718 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.799700 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:14.806195 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:14.809836 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.817303 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.825089 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:14.825991 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.832309 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:14.861224 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:03:14.944969 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:14.945708 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:14.947605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.948518 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:14.951244 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:14.956707 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:14.962867 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:14.998286 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.097592 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:15.098495 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.107574 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.121625 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.127453 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.132627 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.134511 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:15.136833 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:15.149895 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:15.150789 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:15.153438 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.165743 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.188997 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.193786 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:15.197265 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:15.307268 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:15.320367 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:15.330086 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.335480 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:15.336177 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.336987 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.337611 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.357045 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:15.369231 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:15.463032 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.469674 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.481575 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.485544 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:15.495463 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.499563 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:15.502524 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:15.524747 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240502 08:03:15.588917 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.592080 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:15.599025 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.602453 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.607734 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:15.615094 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.615577 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:15.707827 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:03:15.734632 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:15.745539 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:03:15.758003 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:15.816246 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:15.821910 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.834214 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:15.836051 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.837551 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:15.839046 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.841301 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:15.957857 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:15.964012 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.970520 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:15.971696 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:15.986568 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:15.989131 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:15.998713 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.004287 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.046684 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.049965 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.053925 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.057157 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.059800 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.069788 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.073067 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:16.164906 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:16.193637 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.200810 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.202739 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.204308 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.204988 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.208853 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.213761 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:16.216123 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:16.216235 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:16.358340 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.358924 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.377421 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.380098 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.384352 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.384423 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.387516 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.387564 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.393373 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.397071 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.411928 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.434666 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.436074 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.453012 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:16.530223 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:16.556427 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.559011 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.563138 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.564431 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.564942 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.565544 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.567945 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:16.635039 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:03:16.645074 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:16.649875 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:16.694337 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.695443 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.696123 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.696671 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.714232 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.715793 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.731686 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:16.761265 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:16.821242 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.822468 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.828717 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.834553 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.846457 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.849809 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.859324 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.925941 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.929229 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:16.929459 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:16.932667 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:16.942106 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:16.944708 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:16.946419 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:17.015422 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:03:17.016585 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:03:17.029702 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:17.059653 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.059634 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.061388 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.063064 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.067029 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:17.091117 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:17.098320 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:17.098735 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.205076 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.206919 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.218912 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:17.218995 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:17.227864 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:03:17.229594 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:17.242470 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.242620 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.244436 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:17.260622 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:17.291288 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.291522 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.299046 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.300662 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.307227 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.312853 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:17.312961 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.431864 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.436718 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.436714 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.437105 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.461136 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.468806 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.470660 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:17.567353 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.567831 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.572535 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:17.572842 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:17.577031 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.589289 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:17.592200 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:17.602174 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.603345 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.783578 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.785315 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.793462 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.793541 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.809892 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:17.809989 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.823730 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:17.896569 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240502 08:03:17.910012 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240502 08:03:17.929262 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.933370 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:17.945225 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:17.947270 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.947999 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:17.950101 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:17.986311 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:17.987073 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:17.993811 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:17.996904 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.005719 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:18.005868 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.009773 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.013911 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.019125 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:18.048310 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:03:18.079195 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240502 08:03:18.099902 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:03:18.116247 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:03:18.146665 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:18.153769 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.157032 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:18.159041 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:18.159363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.162024 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.169907 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.175856 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:18.304399 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:18.304766 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:18.310703 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.318055 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.334919 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.337648 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.345952 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.360702 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:18.421942 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.423103 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.431356 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.435078 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.454603 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.454946 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:18.473282 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:18.489704 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:03:18.518347 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:18.524003 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.524518 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.526300 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:18.531970 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.533556 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.547194 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:18.554152 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:18.569568 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:18.637284 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:18.664076 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:18.664920 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.672089 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.673995 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.674868 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.705646 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:18.707114 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:18.758630 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:18.796936 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:18.806409 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.807534 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.808045 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.811679 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.811928 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.814617 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:18.887344 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.894026 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:18.896744 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:18.898947 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.899550 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:18.903999 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:18.909287 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:18.990715 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:03:19.010720 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:19.010987 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:19.020088 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.021677 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.021764 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.027405 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.040014 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.043349 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.060132 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:19.150318 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:19.165927 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.176571 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.180076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.194229 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.195942 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.210824 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.231911 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.274405 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.279088 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.283120 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.287959 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.291096 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.319406 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.319736 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.401500 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.402514 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.405434 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.407830 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.422385 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.432559 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.434901 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:19.530894 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:03:19.596338 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:03:19.599680 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:19.627326 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.630980 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.640565 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.640851 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.643283 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.651000 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.654955 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:19.667506 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:19.702033 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:19.755956 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.759125 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.763303 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.764035 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.772086 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:19.789775 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:19.790525 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:19.790616 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:19.795101 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:19.880265 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:19.899334 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:19.902668 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:19.906852 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:19.907356 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:19.914311 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.916065 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:19.918109 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.030007 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.033389 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.039619 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.040602 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.042305 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.052913 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.056411 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:20.064247 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:20.110970 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.113734 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.117520 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.118368 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.125416 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.132658 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.135872 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.245704 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.246456 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.252952 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.260109 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.260195 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.271919 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.272109 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:20.281096 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:20.350205 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:03:20.386894 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:20.398896 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.400553 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.404690 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.420845 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.431018 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.442616 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.443629 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.446161 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:20.452340 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240502 08:03:20.453330 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.476080 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.481026 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.498382 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.499816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.504765 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:20.589625 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:20.610113 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.610939 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.614248 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.623420 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:20.625588 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:20.641433 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.644140 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.656364 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:20.697580 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:20.755172 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.755787 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.759999 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.760711 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.766500 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.789101 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.799443 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.872447 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.872759 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.875751 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.892347 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.896684 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:20.903367 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.906602 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:20.914503 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:20.958403 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:20.961223 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.965574 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:20.966924 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:20.990264 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:20.998279 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.000324 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.106319 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.106705 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:21.108660 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:21.110040 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.110832 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.122346 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.128562 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:21.137184 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:21.202152 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:21.224951 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:21.269560 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:21.271538 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.272141 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.273509 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:21.286999 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:21.291692 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.305382 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.306224 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.308404 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:21.342608 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:03:21.344398 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.347031 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:21.347621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.348528 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.350459 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:21.373553 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:03:21.623325 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:21.657824 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:21.683007 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:21.683431 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.693264 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.695461 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.703171 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:21.706523 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:21.731626 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:21.731928 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:21.732542 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.740615 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.751648 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:21.759295 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.761575 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.776960 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.785456 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.899511 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:21.900807 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:21.903474 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.908005 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:21.910725 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:21.918768 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:21.921010 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.026175 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.034256 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.036430 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:22.037662 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.039817 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.049816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.059091 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:22.076004 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:03:22.079157 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:22.162135 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:22.180824 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.182559 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.187192 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:22.194862 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.197495 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:22.204964 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:22.221545 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.228034 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.234565 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.239495 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.242652 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.248919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.250373 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.261354 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.270912 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:22.293069 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:22.295166 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:22.304584 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:03:22.386332 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.390434 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.391065 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.397989 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:22.404637 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.407292 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.412725 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:22.459786 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:03:22.533805 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.536684 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.549203 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.549355 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.562839 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.565851 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:22.566466 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:22.589803 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:22.631487 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.633008 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.634848 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.645258 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.655120 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:22.670161 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.670290 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.747527 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.748011 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.754943 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.760696 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.765216 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.765293 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:22.779773 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:22.782676 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:22.822793 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:22.843081 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240502 08:03:22.877964 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.883556 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:22.884750 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:22.887602 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:22.887595 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:22.895004 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:22.913156 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:23.005734 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:23.017679 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.029590 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.035102 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.045709 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.047329 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.058624 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:23.073765 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:23.078472 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.083245 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.101677 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.103096 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.110451 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.114629 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.115031 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:23.119248 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:23.123286 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:03:23.128912 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240502 08:03:23.184839 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:23.236001 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.239128 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.239735 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:23.242453 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.248615 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.248811 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.278506 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:23.364514 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:03:23.422915 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:23.424346 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:03:23.426144 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.438802 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.439216 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.441308 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.442087 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.442541 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:23.531255 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:23.576753 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:23.580137 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.581197 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.587213 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.593475 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.612406 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.619695 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.716284 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.719684 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:23.730015 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:23.731539 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.732151 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.733704 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.744690 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.769722 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:23.793530 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:23.835778 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.839488 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:23.844647 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.849298 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:23.859378 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.860633 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.876116 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:23.922281 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:03:23.942242 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:23.945448 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.946161 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.947345 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:23.948137 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:23.950891 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:23.952234 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:23.960388 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240502 08:03:23.996654 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:03:24.013160 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:24.080271 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.085734 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.089773 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.091786 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.096805 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.096899 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.109431 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:24.125705 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:24.222998 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.224013 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.225176 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.241472 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.241482 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.252524 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.271940 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.303186 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.306591 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:24.306999 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:24.310209 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.313186 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.316876 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.319345 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.321048 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:24.429569 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:24.449023 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.452638 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.455101 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.455422 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.465237 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.469000 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.469144 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.587172 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.608330 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.608425 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:24.608804 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:24.608969 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.609897 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.623862 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:24.626219 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:24.629701 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.657097 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.666471 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.667932 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.677129 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.681744 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.688369 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.689894 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:24.696940 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:24.796236 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.798265 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.800033 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.801218 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.820915 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.829173 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.829301 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:24.846345 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:24.918486 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:24.943652 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.943681 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:24.950529 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:24.952890 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:24.959090 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:24.959851 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:24.964810 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.004739 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:03:25.011862 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:25.064594 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.069264 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:25.073390 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:25.079226 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.084045 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.093317 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.094839 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.108523 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.144045 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.144352 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.149957 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.150372 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.155447 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.163281 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.183771 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.278635 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:25.389963 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:25.483865 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.487768 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.488587 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.492395 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.501861 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.521637 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.533946 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:03:25.536644 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.564675 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.567221 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.573194 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.575429 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.577630 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:25.580901 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:03:25.587102 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:25.589059 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.606251 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.680944 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240502 08:03:25.701838 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.708663 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.719185 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.719714 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.720090 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.720846 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.724915 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:25.732949 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.836181 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:25.862682 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.868600 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.871531 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:25.871673 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.879447 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.898532 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.900444 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.901458 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.911005 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.913599 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:25.921383 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:25.930605 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:25.934098 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:25.961493 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:25.975003 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:26.041951 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.046757 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.051353 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.051353 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.061849 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.064718 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.072096 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:26.131760 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:26.181668 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.182808 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.184724 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.185909 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.186847 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.186921 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.197100 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:26.200248 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:26.314553 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.317173 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.332278 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.332329 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.336354 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.338027 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.359622 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.371223 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.375988 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.376542 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.379098 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:26.388140 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:26.389667 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.409807 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.423668 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:26.436558 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:26.528752 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.530364 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.531315 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.541110 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.542887 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:26.545830 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:26.547832 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.558295 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:26.599530 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:26.656039 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.661547 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.663115 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.670959 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.675127 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.675522 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.697209 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:26.721074 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:03:26.760994 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:26.793476 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.799357 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.799916 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.803467 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.806376 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.815583 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.851840 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.887305 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:26.890421 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.892268 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:26.896410 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:26.896757 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:26.916785 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:26.928112 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:27.008427 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:27.026183 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.027351 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.035436 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.036269 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.039533 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:27.041738 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.061158 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:27.091626 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:03:27.092603 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:03:27.115006 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:03:27.164009 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.168215 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:27.170981 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.178607 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.179515 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:27.194129 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.194175 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:27.284286 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:27.363857 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:27.368261 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:27.369380 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.371886 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.371946 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.376633 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.380853 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:27.397699 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:27.501324 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:27.524786 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.530436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.531081 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:27.538875 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.555145 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:27.559041 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.560318 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.563263 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.579104 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.581487 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.589058 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.592644 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:27.596982 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:27.630460 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:27.633858 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:27.715934 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:27.735849 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.736091 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.736704 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:27.738153 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.749586 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.752517 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:27.765476 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.887915 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.890321 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.891036 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:27.892586 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:27.896479 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:27.897740 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.899865 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:27.994262 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:27.997926 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.000597 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.001559 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.005584 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.010061 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.028996 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.050076 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:28.084353 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.085436 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.087095 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.097312 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.102038 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.104698 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.112699 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:28.121850 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:28.168536 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:03:28.169651 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240502 08:03:28.184937 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240502 08:03:28.186549 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:28.219309 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.220388 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.221890 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.231531 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.232007 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.232357 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.241415 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:28.251914 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240502 08:03:28.266811 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.359349 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.365208 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.365721 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.366760 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.369872 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.393687 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.401311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.434877 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.437760 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.440820 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.441267 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.442634 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.452409 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.466603 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.510031 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:28.562574 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.565505 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:28.567709 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:03:28.569998 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.572649 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.575296 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.577783 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.585394 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.629786 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:28.700492 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.701283 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.709260 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.709873 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.714238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.714336 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.756692 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.781767 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:28.803254 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.807662 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.810326 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.817159 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.824674 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.828197 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.841133 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.863400 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:03:28.912652 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:28.927420 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:28.927431 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.927759 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:28.928094 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:28.928332 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:28.939154 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:28.957890 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:28.997640 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:03:29.080585 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:03:29.107084 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:29.150302 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.156400 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.157238 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.159219 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:29.160339 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.161592 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.169909 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.279286 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.279700 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:29.280582 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:29.282509 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:29.285072 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:29.286901 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:29.298023 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.298341 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.311100 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:29.369844 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:29.413260 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.413566 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.417634 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.417692 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.431893 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.431893 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:29.432768 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.534329 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.537104 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.537119 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.548648 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:29.548957 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.552383 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.561400 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:29.593138 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:29.617991 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.621879 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.623006 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.623494 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.634678 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.638641 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.639117 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:29.693634 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:29.760623 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.764020 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.764276 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.764660 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:29.772030 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.782377 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.788679 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:29.845707 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:29.853319 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:29.897991 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.901623 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.903431 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.907335 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:29.912408 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:29.919719 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:29.929962 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:29.936328 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:29.992092 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:29.992779 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:29.997637 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.017517 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.018496 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.020781 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.029278 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:30.112552 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:30.129397 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.130070 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.130796 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.143278 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.148677 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.150761 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.175760 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.264320 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.269620 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.272909 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.273109 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.273321 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.281029 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.285598 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:30.312873 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:03:30.323886 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:30.345921 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.351528 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.355366 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.361922 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.361935 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.362493 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.372757 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:30.400851 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:30.435968 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:30.481931 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.487697 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.488190 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.488319 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.489523 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.500322 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.516749 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:30.599968 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:30.628446 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.640270 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:30.644673 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:30.647827 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.648633 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.652735 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.656706 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.677954 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.714613 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.718623 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.722621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.731775 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.744108 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.744371 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.763962 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:30.800062 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240502 08:03:30.840765 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:30.844727 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:30.850615 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:30.851845 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:30.867585 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:30.871834 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:30.872404 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:30.873842 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:30.948050 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:31.012255 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:31.072072 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.077934 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.078650 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.078694 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.079646 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.096953 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.096943 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.205101 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.205958 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.209388 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.209549 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.211519 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.216675 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.218297 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.337687 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.342584 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.345512 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:31.349298 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:31.349907 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.350155 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.355612 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.371904 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:31.435076 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:03:31.480427 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.487377 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.488960 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.495162 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.505092 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.523150 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.525323 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:31.554231 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:31.555349 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.565658 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.567365 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.567385 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.571520 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.571928 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.583387 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:31.644887 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:03:31.705938 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.710161 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.710278 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.720628 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.724901 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.726449 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.726786 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:31.759002 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:31.770083 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:03:31.826570 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:31.826663 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:31.862998 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.867642 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.873365 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.879791 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.888725 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.894363 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.898312 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.898669 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.909351 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:31.923547 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:31.931439 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:31.935493 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:31.939361 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:31.962288 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.068778 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:32.071013 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.071396 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.073854 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.085436 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.093398 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.093894 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:32.113967 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:32.216895 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:32.230198 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.230515 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.231389 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.231666 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.240706 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:32.245489 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:03:32.272116 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:32.284018 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:32.293277 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:32.293680 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:03:32.329891 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.330293 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:32.339674 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:32.340579 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.343089 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.353111 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.357223 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.383613 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:32.430691 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:03:32.441661 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:32.447389 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.448208 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.452929 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.457655 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.473304 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.493319 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.590446 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.590526 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:32.596518 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.605862 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.609483 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.613996 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.644177 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:32.652997 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:32.715922 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.723670 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.732575 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.734185 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.736984 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.755825 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:32.759847 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.820580 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:32.820657 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:32.821100 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.833714 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:32.838011 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:32.840514 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:32.853212 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:32.854557 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240502 08:03:32.949769 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:33.065297 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.068761 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:33.068769 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:33.069082 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.077976 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.080772 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.111586 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:33.135007 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:33.150367 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:33.176939 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:33.183111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.187951 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:33.190352 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.194448 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.199930 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.200243 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.216658 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:33.259616 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:03:33.319468 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:33.321794 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.332989 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.334024 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.339845 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:33.352749 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240502 08:03:33.353174 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.359313 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:33.406860 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:33.452378 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240502 08:03:33.474771 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.479713 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.487901 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:33.497498 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.499001 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.502278 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.509560 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:33.539304 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.540409 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:33.544771 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.545085 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.545279 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.574482 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:33.581391 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:03:33.585536 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:33.607203 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:33.691334 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.692828 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.696178 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:33.696899 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.697017 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.713119 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:33.735918 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:33.744446 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:33.848963 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.851435 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.855986 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.865916 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:33.867312 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.882489 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.883220 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:33.901700 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:33.902734 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.916060 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:33.917280 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.939643 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:33.941005 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:33.960567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:33.969964 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:34.023175 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:34.053458 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.058422 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.058501 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.064332 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:34.074762 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:34.075537 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:34.079692 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:34.081422 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:34.155943 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:34.198320 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.202458 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.209300 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.210124 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:34.210515 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:34.232687 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.232815 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:34.291254 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:34.326987 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.330363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:34.338558 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.339795 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.344076 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:34.357321 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.365098 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.414414 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.417032 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.425134 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:34.431427 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.431950 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.441555 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.442485 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:34.443383 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:03:34.491467 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:34.504240 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:34.533376 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:34.553859 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:34.558580 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:34.559468 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.564328 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.567319 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.576054 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.577577 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:34.586409 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.711833 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.714969 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.722895 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.726400 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:34.739107 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:34.745127 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.754201 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.755157 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:34.755465 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:34.759742 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:34.765448 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.771730 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:34.784785 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:34.813983 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:34.819034 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:34.862239 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:03:34.943718 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:35.003962 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240502 08:03:35.026815 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:35.130031 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.133028 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:35.140662 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:35.140659 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.151621 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.153973 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.154538 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:35.229360 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:35.237910 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.238025 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:35.240764 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.242700 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.243551 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:35.265416 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.389487 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.389770 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.397063 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:35.400732 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.407894 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:35.410403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:35.413105 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:35.417410 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:35.425160 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:35.448491 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:35.489972 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:35.532346 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.543711 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.544704 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:35.549924 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.566330 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.573956 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:35.578351 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:35.592294 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:35.596944 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:03:35.600996 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:35.604032 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:35.606343 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.620296 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.621066 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:35.625551 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.636700 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.769748 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.774084 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:35.774092 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:35.776496 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.785081 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:35.802534 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.806186 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:35.835604 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240502 08:03:35.882337 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:35.883770 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:35.921141 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.921141 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:35.922469 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:35.925830 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:35.933233 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:35.937218 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.939926 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:35.940373 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:35.953475 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.017513 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.017879 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.022790 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.027968 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.031339 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.052301 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.068641 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.115635 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.118769 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.118853 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.121640 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.123862 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.136384 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.137240 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.264447 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.265825 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.268574 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.272401 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.279486 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.289927 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.321249 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:36.337677 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:36.385051 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.388334 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.392625 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.401520 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.407264 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:36.411718 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:36.412667 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:36.439574 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:36.442896 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.489362 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.491963 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.495199 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.495642 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.496340 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.510589 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.515130 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:36.610666 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:36.627270 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:36.628108 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:36.634581 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.635598 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.635857 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.638383 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.650149 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.652768 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.782318 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.782969 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.785120 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.785564 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:36.804229 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:36.815521 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:36.820722 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.850984 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.859251 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.859563 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:36.864073 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:36.866569 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:36.874800 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:36.875604 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:36.885572 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:36.887884 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:36.901574 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:37.028309 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:37.212782 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.221236 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.224514 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.231648 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.233105 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:37.243732 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.263228 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:37.264067 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.268713 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.278656 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.280797 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.286062 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.296854 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:37.312952 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:37.316303 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:37.320083 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:37.374382 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:03:37.415266 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:37.422097 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.422490 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.423847 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.431602 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:37.440529 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:37.443565 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.443658 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:37.480077 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:37.553961 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:37.579705 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.580976 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:37.586144 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.595364 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.597760 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.611157 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.611673 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:37.685305 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.688146 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.689985 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.691779 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.701786 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:37.710191 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:37.722982 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:37.741261 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:37.750572 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:37.788296 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.788292 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.788733 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:37.793388 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.794806 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.811347 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:37.811808 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:37.928853 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:37.935695 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:37.939334 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:37.941972 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.942593 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:37.942703 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:37.950132 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:37.958231 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:37.985860 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:38.038189 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:38.056231 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.064023 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.066692 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.071774 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.074458 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.079885 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.086527 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.147675 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.153782 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.154091 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.154680 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.154925 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.167313 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.188278 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.286545 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.287966 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.294158 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.296556 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.300331 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:38.305343 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:38.305760 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.309073 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:38.351121 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:03:38.421408 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:38.427575 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.429358 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.435890 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.459856 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.467352 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:38.470449 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:38.478890 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.484272 28770 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.484617 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.485055 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.502872 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.505244 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.512029 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:38.514644 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240502 08:03:38.516661 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.548162 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:38.562232 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:38.646421 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.651458 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.654186 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.658602 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.665668 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.666343 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.682853 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:38.703174 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 08:03:38.703984 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:38.793995 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.800027 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.805840 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:38.807111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.813221 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.813663 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.814447 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.895354 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.897843 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:38.900429 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:38.904654 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:38.913347 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.913437 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:38.936738 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:39.012933 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:03:39.117944 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:39.144028 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.143579 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:39.145440 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:39.147444 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:39.151358 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.159070 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.159770 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.204293 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:39.216799 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:39.264281 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.271327 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:39.282676 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:39.283864 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:39.285770 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240502 08:03:39.299685 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:39.307518 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.309612 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.310803 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.368788 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:39.373140 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.376320 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:39.380645 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.385643 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.392572 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.398999 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:39.487496 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:39.505841 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:39.506048 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.510535 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:39.516716 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.527963 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.532914 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.534147 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:39.557600 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:39.657363 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.659149 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:39.663832 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.667579 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.677685 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:39.690024 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.691046 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.691710 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.696480 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:39.717965 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:39.725651 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.731303 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:39.735561 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.740267 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.764251 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:39.772392 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:39.860867 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.862000 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:39.868894 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.869032 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:39.879060 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:39.879593 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:39.879717 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:39.922470 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:03:39.923327 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:39.999223 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.008601 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.009366 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.010090 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:40.021613 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:40.029157 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.029891 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.039590 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:40.111428 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:40.123814 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.125576 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.126483 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.126765 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.132009 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.133627 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:40.141287 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.207295 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.211678 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.214354 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.215881 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.220579 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.226081 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:40.230252 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:40.245653 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.349669 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.351478 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.354761 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.358812 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.360769 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.371513 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:40.383368 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.486922 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.492554 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.500270 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:40.501534 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:40.513178 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:40.521458 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:40.526986 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:40.530040 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.537746 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.544013 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.554222 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.555147 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:40.563783 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:40.568666 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:40.575894 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.595252 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.602310 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:40.644560 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:03:40.708619 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:40.709576 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.710245 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.711975 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.713485 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.713604 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.718776 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.731366 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:40.746212 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:40.895941 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.897068 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:40.898743 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:40.901917 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:40.902117 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:40.902987 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:40.926200 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.026616 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240502 08:03:41.043987 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.048754 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.051076 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.052539 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.055279 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.063481 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.070513 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:41.075132 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:41.087715 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:41.100809 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:41.193156 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.200845 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.202942 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.218228 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.224403 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.224628 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.225208 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.228729 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:41.256914 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:41.312170 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.314626 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.314777 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.319542 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.340692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.345975 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.347318 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.405185 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.407392 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.408139 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.411010 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.419772 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.421629 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.433252 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.479480 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:03:41.491175 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:41.540793 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.540787 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.541291 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.543258 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.567770 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.569730 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:41.574005 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.575693 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.620297 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:41.677397 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.682898 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.686928 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.701102 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.710072 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.719266 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.722514 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:41.752444 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:41.771323 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.774927 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.783128 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.785030 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.787986 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.791219 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.792174 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.915773 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:41.921638 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.922065 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:41.927248 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:41.928022 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:41.931265 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:41.942992 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:42.048972 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:42.051473 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.054540 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:42.059520 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.064329 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.073325 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.073328 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.106319 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:42.108821 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:42.110702 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:42.119982 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.134168 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:42.138239 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.138321 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:42.141739 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:42.146606 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.173267 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:42.181862 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:42.257483 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.258684 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:42.267369 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:42.272645 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.276314 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:42.283216 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.294036 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.294199 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:42.395661 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:42.399991 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.408422 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.409569 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.411780 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.414561 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.415426 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:42.417312 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:42.484938 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:03:42.499889 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:42.501466 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.502475 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.512712 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.515951 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:42.524621 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.535933 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.591172 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.594013 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:42.595686 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.599257 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:42.599337 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.605360 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.615491 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:42.648051 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:42.694998 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:42.863401 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.866657 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:42.866722 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:42.867833 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.868984 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:42.870887 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.872761 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.982544 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:42.983225 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.983230 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:42.983605 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:42.996714 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.005661 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.007652 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:43.044219 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:43.063573 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:43.108915 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:43.113396 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.117569 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.124799 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.126405 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.130055 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.155292 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:43.170557 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:03:43.201727 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:43.247570 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.254597 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.257313 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.258836 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.260890 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:43.264186 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240502 08:03:43.272766 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.280874 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:43.316754 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:43.328631 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.330138 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:43.340378 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.342434 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.344062 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.365990 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.373759 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.470269 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:43.473088 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:03:43.482560 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.484294 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.484604 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.487025 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.501291 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.512323 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.614111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.617724 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:43.620823 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.623100 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:43.655766 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:43.657078 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.658011 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.659613 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.707536 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:43.708568 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:03:43.709338 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:43.712277 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.715510 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:43.721477 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:43.722244 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:43.727883 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:43.739008 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.742321 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.757628 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.817274 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:43.821647 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.822520 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:43.825022 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:43.836524 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.841871 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.853526 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.857915 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.957930 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:43.962908 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.966506 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:43.967725 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:43.969442 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:43.991379 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:43.995767 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:44.054068 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:44.098400 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:44.103809 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.108146 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.115666 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:44.116415 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.121052 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.124604 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:44.153483 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:03:44.160614 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:03:44.164582 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:44.166605 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:44.170786 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.183260 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.185896 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.198139 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.203864 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.219286 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:44.239626 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:44.296428 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.299726 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:44.302364 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.303490 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.304332 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.305652 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:44.310731 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.431725 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.432287 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:44.433838 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:44.437372 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.437799 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:44.443912 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.450886 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.466727 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:44.510491 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:44.558707 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.561070 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:44.564837 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.565124 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.565426 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.570353 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.597709 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:44.650516 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:44.755440 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:44.768621 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.774595 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.775161 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:44.776818 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.784881 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:44.785555 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:44.816326 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:44.826047 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.905193 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.905297 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:44.913573 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:44.916966 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:44.929459 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.932961 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:44.965538 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:44.997233 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:45.000761 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:45.001317 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.005921 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.005897 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.010948 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.011422 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.011960 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:45.043329 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:45.133363 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.133507 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.137281 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.147891 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.149540 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.151216 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.151464 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:45.207468 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:45.279242 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.292534 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.294778 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.308121 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.311062 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:45.312284 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:45.319558 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.332620 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.345001 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.346786 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.359973 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.360386 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.362697 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.370640 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.388959 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:45.451431 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:45.465507 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:03:45.473214 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240502 08:03:45.490844 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.492300 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.493778 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.497825 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.499758 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.508648 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.537235 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.628158 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.629668 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.629832 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.631223 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.639669 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.657953 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.664500 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:45.688171 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240502 08:03:45.743315 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:45.760246 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.762436 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.765153 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.775427 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.776659 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.780025 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:45.798120 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:45.800253 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.846125 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:45.846305 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:45.852859 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:45.854557 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.856951 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:45.863848 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:45.877089 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.892622 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:45.902060 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:45.977207 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.984318 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:45.985965 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:45.998201 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.000097 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.009168 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.029510 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.121474 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:46.123354 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.134361 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.138283 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.144018 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.151556 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.163542 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.182499 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.188082 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.188359 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:46.207849 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:46.211400 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:46.211786 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.222321 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.225019 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:46.253155 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:03:46.306560 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:46.338493 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:46.340981 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.351173 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.353435 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.354409 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.356812 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.359781 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:46.371006 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:03:46.431687 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:03:46.514343 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:46.515547 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:46.516263 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.523355 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.523356 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.531530 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.537653 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.542771 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:46.654000 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:46.680531 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.687168 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.689637 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.691778 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.692384 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.692972 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:46.713301 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:46.724489 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:46.732776 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:46.838255 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.839888 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.848596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:46.857570 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:46.859660 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.859887 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:46.867998 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.872987 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.905053 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:46.919849 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:46.922091 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:46.929286 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.931048 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:46.941687 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:46.945101 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:46.946904 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:46.995853 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:47.042953 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.043088 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.049679 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.053519 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.060482 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.061178 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.075172 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.174595 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.177995 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.180502 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.185181 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.194172 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.201504 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.204605 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:47.258019 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:47.309831 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.318744 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.319450 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:47.320320 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:47.320984 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.333835 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.336982 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.347553 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:47.393455 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:47.398445 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.398545 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.398681 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.401659 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.402647 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.405126 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.413422 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:47.494933 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:03:47.529204 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.531766 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.532682 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.534157 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.535825 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.546875 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:47.547334 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:47.557983 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.679741 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.682291 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.685143 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.691269 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.692984 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.696242 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.702178 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.717108 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.722430 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.725106 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.733531 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.749640 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.762935 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.773180 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:47.800446 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:47.825399 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:03:47.870955 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:47.871443 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:47.880857 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.885087 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:47.887864 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:47.888373 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:47.894621 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:47.914742 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.013630 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:48.015581 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240502 08:03:48.016103 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.019881 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.020685 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.027937 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.030634 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:48.060685 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:48.076061 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:48.130875 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.140269 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.140771 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.141644 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:48.143707 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.148633 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.148651 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.221354 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.224303 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.224720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.225672 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.228775 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.230149 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.246192 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:48.304715 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240502 08:03:48.410516 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:48.457623 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:48.494815 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.495124 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.495724 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.496517 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.497942 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.520946 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:48.524597 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:48.579138 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:03:48.605132 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.610714 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.613667 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.616933 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.616981 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.623587 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:48.626801 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:48.644907 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:48.746814 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.753707 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.757324 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.757691 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.757812 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.760264 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:48.783169 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:48.847636 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:03:48.880501 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.880707 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.883379 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.888303 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:48.888693 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.892890 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.926707 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:48.929350 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:48.965653 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:48.967226 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:48.972574 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:48.974143 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:48.976111 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:48.984442 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:48.989763 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.009713 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:49.050184 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:03:49.052259 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:49.090579 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:49.107004 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.108309 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.108840 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.109619 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:49.122287 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:49.126628 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.127104 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.133591 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:49.228281 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:49.246464 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.246655 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:49.248782 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:49.250131 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.272548 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.272562 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.289395 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:49.303915 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.305974 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.324254 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.338270 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.343609 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:49.345821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.348196 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.348511 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:49.389982 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:49.416922 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:49.465816 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.467875 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.478377 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.483709 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.485538 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:49.489370 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.491166 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.624346 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.631350 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.634332 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.636520 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.641882 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.653617 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.656771 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:49.671330 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.676342 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.678552 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:49.679692 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:49.701404 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.716285 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.719817 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.723907 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:49.751914 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:49.820116 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.820643 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.827314 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:49.827384 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:49.835072 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.838109 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.868800 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:49.939371 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:49.976876 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:49.981701 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:49.986007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.992034 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:49.992311 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:49.994344 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:50.002606 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.100014 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:50.103622 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.109833 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:50.114176 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:50.116062 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:50.140231 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.150480 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:50.157866 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.166599 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.175609 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.181967 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.182780 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:50.186959 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:50.204845 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:50.210587 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.214099 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:50.276594 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:50.384029 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:03:50.390416 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240502 08:03:50.454034 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:50.518560 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:50.533691 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:50.533936 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.537932 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.541442 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.546669 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.561486 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:50.587828 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:50.591619 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:50.593891 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.610709 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.617508 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.617776 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.618183 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:50.652190 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:50.683140 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240502 08:03:50.739512 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:50.739899 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.740953 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:50.741115 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:03:50.751436 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.753237 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:50.771071 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.778191 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:50.863240 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:50.883116 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.883435 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:50.889328 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.890595 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:50.894379 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:50.894441 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:50.909281 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:50.992573 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:50.999657 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.001747 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.012789 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.013597 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.018662 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.034865 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.092399 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:51.094516 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.096531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.097198 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.102602 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.115958 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.130136 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:51.168205 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:51.179601 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:03:51.202090 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:51.234980 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.235294 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.237126 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.237159 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.239542 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.249625 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:51.271081 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:51.359532 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:51.369652 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:51.370941 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.375948 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.376664 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.389964 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:51.390367 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:51.393867 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:51.405751 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.407492 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.449800 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:51.449913 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.458546 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.462291 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.481278 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.482033 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.493120 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.589711 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.591233 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.592952 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.603328 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:51.603603 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:51.604261 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.605952 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.608984 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:51.676090 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240502 08:03:51.716559 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:03:51.725181 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:51.742296 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.742889 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.744242 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.757781 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.760354 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.765682 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.792716 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:51.806743 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:51.846838 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.848603 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:51.861533 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.865952 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.866508 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.888586 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.899716 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.963627 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:51.964159 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:51.967561 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:51.969643 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:51.973744 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:51.980717 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:51.982110 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:51.983626 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:51.985284 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.095999 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.100437 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.102394 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.104641 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.107980 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.115187 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.118294 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:52.143872 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:52.215888 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.217859 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:52.221890 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:52.223798 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.224546 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.230412 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:52.239426 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.240819 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.423725 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.431512 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.431965 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.432194 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.432761 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:52.433704 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.435029 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:52.550403 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:52.572330 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:52.573828 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:03:52.574643 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:52.580243 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:52.582160 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.585430 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.594172 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.605336 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.606601 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:52.613889 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:52.614789 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.615944 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.625991 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.627583 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.628156 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.639171 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:52.657303 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:52.757057 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:52.782157 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.782270 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.782666 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.784278 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.789624 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:52.800662 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.803030 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:52.900010 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:03:52.901160 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:52.921087 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.924180 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:52.931434 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.933449 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:52.933962 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:52.940606 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:52.942324 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:53.012914 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:53.046212 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.047466 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.050673 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.056025 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.067066 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.069334 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.081269 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.128818 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.131232 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.131897 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.134770 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.143020 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.152750 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.161984 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:53.166851 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:53.260205 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.263664 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.273510 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.275844 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.285866 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.286731 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.311360 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:53.358026 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240502 08:03:53.399979 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.405354 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.411971 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.412227 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.424686 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.441665 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.442095 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.488185 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.490396 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.495537 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.502610 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.503926 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.509256 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.526548 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:53.544040 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:53.614213 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.617431 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.624190 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.624670 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.624852 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.628369 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.660291 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:53.663046 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:03:53.664968 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:03:53.766940 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:53.781013 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.782562 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.783383 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.784636 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:53.785128 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:53.792928 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.803318 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.803974 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.808588 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.828413 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.841315 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:53.841683 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.849397 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:53.877907 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.880923 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:53.920219 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:03:53.921196 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:53.945994 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:53.984757 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:53.986827 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:53.987721 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:53.996866 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:54.005653 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.015092 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.015847 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:54.093341 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:03:54.105923 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:03:54.161715 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:54.187106 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.189481 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.197041 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:54.198930 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:54.201720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.215874 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.216192 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.324342 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.325773 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:54.327955 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:54.329592 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.336933 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:54.338939 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:54.345566 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.361117 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.363552 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:54.372153 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:03:54.374245 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:54.465665 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.470850 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.477361 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.483371 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.483634 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:54.491322 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:54.501686 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.575057 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.576653 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.580142 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.584393 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:54.610062 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:54.610603 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.611708 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.674039 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.674508 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.676808 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.680016 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:54.680657 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.683974 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:54.702391 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:54.760200 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:54.814356 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.814922 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.820318 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:54.824627 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:54.829806 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:54.850529 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:54.851058 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:54.854671 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:54.855973 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:54.881958 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:54.962855 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:54.964881 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:54.969183 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:54.970710 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:54.977190 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.000968 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.014626 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.015462 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.016270 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.018216 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.026968 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.032290 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.057271 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.077401 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:55.152711 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:55.173017 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.179257 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.181003 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.181995 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.182144 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.196507 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.200047 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:55.305512 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:55.309505 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.312708 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.317143 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.320039 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.326956 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.337482 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:55.338455 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240502 08:03:55.340692 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:55.416997 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:55.428215 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.429225 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.429828 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.431617 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.436707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.465262 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.489794 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.493220 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.494997 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.513900 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.518414 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.519778 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.545842 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.559320 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:55.601790 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:03:55.632841 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:55.657828 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.659413 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.662078 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.666625 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.671803 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.693614 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.693965 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.798931 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.799629 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.805316 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.809721 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.820664 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.820972 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:55.821233 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240502 08:03:55.831907 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:55.847925 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.890012 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:55.894109 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.905941 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:55.906608 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:55.911144 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:55.925662 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:55.928122 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:56.033146 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:56.125787 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240502 08:03:56.152501 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:56.175207 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.176002 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.176716 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.184729 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.187981 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:56.188766 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:56.224066 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:56.263084 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:03:56.272102 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240502 08:03:56.290058 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.296260 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.297344 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:56.298110 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:03:56.301036 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.306188 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:56.306473 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.306738 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:56.379562 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:56.389047 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.390429 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:56.393364 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:56.402194 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.404662 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.407768 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.408934 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:56.431345 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:56.488154 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:56.527683 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:56.528115 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.529270 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.529510 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.545255 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:56.545876 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.547556 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:56.568753 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:03:56.613070 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:56.653373 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.655725 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.659255 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:56.661682 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.667869 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.670923 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:56.681910 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.728231 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.732784 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.739784 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:56.745750 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.763813 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:56.766181 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:56.768062 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:56.772588 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:56.784162 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:56.809410 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:56.884780 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:56.890472 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.892411 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.895718 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:56.898015 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:56.901882 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:56.903944 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.023465 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.023545 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:57.025089 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.025579 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.033488 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:57.061472 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:57.062772 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.080930 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.134797 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.140264 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.145330 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.148980 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.156297 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.162319 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:57.169121 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:57.243396 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:57.257921 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.259297 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.271353 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.272034 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.273977 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.287744 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.289104 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:57.300837 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:57.392182 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.393975 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.398759 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.400578 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.403879 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.405232 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.406801 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:57.519551 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.524940 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.531886 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.535630 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:57.540962 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.544596 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.547026 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.584265 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.586617 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.587937 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.593991 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:57.600541 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:03:57.606659 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240502 08:03:57.621778 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:57.624915 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.633870 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.645751 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:57.713402 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:57.757640 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.763927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.765801 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.768198 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:57.779423 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.779942 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.792869 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.920794 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.921154 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:57.921615 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:57.928225 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.933744 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:57.948520 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:57.966321 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:57.974570 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:03:58.055423 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:58.088991 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.092036 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.094928 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.095343 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.095574 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:58.109544 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:58.114161 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:58.116705 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:58.128978 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.228062 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.235146 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:58.237942 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.239539 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.242385 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.243444 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.252665 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:58.255972 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240502 08:03:58.280568 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:58.329663 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.332396 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.332561 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.340095 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.362280 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.368028 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:03:58.369288 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:58.386883 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.434939 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.437291 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:58.441659 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.441737 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.457662 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.462987 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.471140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.579443 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:58.579663 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.579778 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.592056 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.592387 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.592806 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.617941 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:58.646981 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:58.665696 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:58.709619 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:03:58.709812 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:58.711591 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:03:58.713049 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.725349 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.730372 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.731719 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:58.733039 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.753351 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.797183 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.798488 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:58.802670 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.806550 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.811231 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.811728 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.826967 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:58.848798 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:03:58.892025 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:03:58.939301 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:58.942061 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.945430 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:58.945963 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:58.957280 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:58.957365 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:58.960832 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:58.989668 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240502 08:03:59.056404 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:03:59.058745 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:03:59.070272 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.077906 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:59.079047 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.081023 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:59.089537 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.090637 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:03:59.102102 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:03:59.120160 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.141865 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.142841 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.144176 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.154506 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:03:59.173409 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:03:59.186837 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.191645 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:59.200089 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:59.201829 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:59.283742 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:59.298426 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.301810 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.301872 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.302606 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:59.311025 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.319505 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.320174 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:59.442970 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.443663 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.458410 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.458464 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:59.458732 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:59.486449 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.489681 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.516872 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.524720 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.529371 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:59.530503 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.532477 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:59.548928 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.568127 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:03:59.593796 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:03:59.649276 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.649648 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.657042 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.657620 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:59.658149 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.660499 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.661319 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:03:59.780750 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:03:59.820906 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:03:59.874619 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:03:59.877266 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:03:59.877826 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:03:59.878304 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.886739 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:03:59.888211 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:03:59.920812 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.001341 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.001426 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.001827 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.006167 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.006794 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:00.007265 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.027550 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:00.044131 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:00.131327 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.132972 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.139353 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:00.141340 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.141722 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.143239 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.148640 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:00.153779 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:04:00.277967 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.282625 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.288597 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.294432 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.312186 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.316207 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.318120 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.324834 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.333287 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:00.346464 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:00.353581 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.355196 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.357969 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:00.371430 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.375913 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:00.386134 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:00.431335 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:00.483892 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.484678 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.485126 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.491875 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:00.493561 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.518409 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.530277 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.629509 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.634786 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.635811 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.635937 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.641300 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.660135 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.660849 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:00.666309 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:00.764875 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.765128 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:00.779126 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:00.781040 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.786054 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:00.786691 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.788427 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.802392 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.851684 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:00.855537 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:00.856321 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.860805 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:00.863843 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240502 08:04:00.866855 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:00.874946 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:00.898586 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:00.964566 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:01.000002 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:01.000474 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:01.002575 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:01.011020 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.012643 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:01.014953 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.015007 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.029148 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:01.121958 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:01.142172 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:01.142675 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.154135 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.155723 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.158136 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.175227 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:01.192281 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:01.205674 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.209156 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:01.218335 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:01.222208 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.228811 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.229843 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.236675 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.239957 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:01.260313 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:01.261144 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:04:01.285111 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:01.345170 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:01.348587 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.351608 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:01.356364 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.360894 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.361814 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.382195 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:01.413553 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:01.487093 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.494153 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:01.495095 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.500695 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:01.503954 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:01.505228 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.510924 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.523674 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:01.538633 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:01.558149 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:01.606817 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:01.614708 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.615490 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.617961 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.626389 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:01.636350 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.640347 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:01.789412 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:01.790201 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:01.834149 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.834815 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:01.841306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:01.846984 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:01.850445 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:01.852753 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:01.856364 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:01.949501 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:01.999068 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.002113 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.005841 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.009728 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.018088 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.020658 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.057215 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.057623 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.058218 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.058677 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.062330 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.079905 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.080190 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.097785 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:02.117741 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:02.205921 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.212126 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.216151 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.216665 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.217142 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.218869 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.225697 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:02.236807 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:04:02.339500 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:02.348575 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.349356 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.352423 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.354158 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.364758 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.368029 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.389834 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.462967 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:02.466025 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:02.469259 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.473665 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.485991 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.496560 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.497428 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.499322 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.553572 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.555363 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.560189 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:02.566565 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:02.567276 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.569200 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.580983 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.581944 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:02.621667 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:04:02.652572 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:02.691787 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.693549 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.695382 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.702467 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.704314 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.710294 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.716895 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.834431 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.836549 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:02.836720 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.844894 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.847465 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.856879 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.871855 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.917590 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:02.917690 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.921259 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:02.921273 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:02.924880 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.925355 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:02.948942 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.038914 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.041908 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:03.043838 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:03.044366 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.047791 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.047792 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.056713 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:03.058192 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:03.069876 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.182479 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.183542 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.189253 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.193763 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:03.193872 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.194675 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.236989 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.238770 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.239612 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.246493 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:03.250986 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:03.254563 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:03.254642 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:03.268357 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:03.269122 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.281244 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.283924 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:03.355654 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:04:03.358651 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240502 08:04:03.385445 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.385653 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.389465 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.397496 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.401084 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:03.412818 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.417613 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:03.461385 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:03.490944 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:04:03.539067 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:03.590963 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.593447 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.593880 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.595382 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.601127 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:03.608994 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.625041 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:03.630103 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:03.721221 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.722625 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.723371 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.723863 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.724975 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.725567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:03.745029 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:03.765302 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:03.861831 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.862300 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.867435 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.876688 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:03.876823 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:03.886247 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:03.904309 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:03.929601 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
I20240502 08:04:03.986164 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:03.988744 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:03.990212 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:03.996618 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:03.996757 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:03.999413 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.009876 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.020287 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:04.040324 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:04.080353 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.082433 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:04.084163 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:04:04.085325 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:04.087172 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.088168 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.094345 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.110905 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.111511 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.211057 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.213136 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:04.215220 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:04.217767 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.218802 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.225790 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.227826 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.233490 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:04.327535 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:04.364356 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.370961 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.377889 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.381309 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.386871 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.390885 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.409080 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.411839 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.417263 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.419901 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.433661 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.434362 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.434536 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.463725 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:04.483096 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:04:04.487843 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:04.566663 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.568693 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.571298 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.578642 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.581173 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.584280 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.584622 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.696630 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.697871 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.699064 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.700861 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.712823 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.721311 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.738380 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.824791 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.826931 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.829133 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.832067 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.841081 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.845170 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.846163 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:04.890519 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:04.892509 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:04.893754 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.895767 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:04.907486 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:04.922858 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:04.926208 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:04.931900 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:04.992317 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:05.030779 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:05.045981 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.048704 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:05.049293 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.051489 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.053192 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.054324 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:05.073786 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:05.177942 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240502 08:04:05.179208 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:05.179589 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.188655 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.193738 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.194885 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.198153 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:05.205142 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.286164 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.287343 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.296000 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.296789 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.299122 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:05.304700 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.325588 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:05.351972 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:05.468286 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:05.522646 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.524442 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.525319 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.528371 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:05.540076 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:05.548071 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.550793 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:05.577920 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:04:05.578015 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:05.670228 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.670855 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.671829 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:05.682500 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.683311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.693949 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.699137 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:05.713106 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:05.721369 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.732448 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.755718 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:05.762159 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:05.776051 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.778209 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.779758 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:05.792299 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:05.887197 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.889820 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:05.896392 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:05.896658 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:05.899312 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:05.900135 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:05.908347 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:05.985973 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:04:05.996332 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240502 08:04:06.007408 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:06.011323 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:06.027731 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.028874 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.039570 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.039594 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.047171 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:06.050628 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.051040 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.107897 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.108604 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.114732 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.120797 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.130805 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:06.138136 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:06.144829 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:06.149138 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.154783 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.211122 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.213492 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.217548 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.219154 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.221701 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.225075 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.228375 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:06.319633 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:06.349298 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.353068 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.354101 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.356226 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:06.359232 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.374833 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.389182 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:06.454656 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:06.491997 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.501052 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.502380 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.505491 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.505908 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.508417 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:06.529736 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.535115 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.542155 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.543414 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.561300 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.565003 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:06.577757 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.585606 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:06.608520 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:06.623392 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:04:06.687093 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.687922 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.691099 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.693279 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.698220 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.700807 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.724674 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:06.748582 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:06.758977 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:06.834996 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.835731 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.837301 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.844616 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:06.847179 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:06.849876 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:06.859400 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:06.874830 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:06.880106 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:06.969491 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.972373 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:06.975221 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:06.980571 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:06.998617 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.010946 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.012861 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.014621 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.016357 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.028492 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.038878 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.048537 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.051311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.065922 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:07.205332 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:07.320422 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:07.332777 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:07.382007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.387061 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.388177 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.389776 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.392880 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.420713 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.430794 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.431236 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.431744 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.436508 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.448282 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.473316 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.477797 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.478411 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.576985 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.577507 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.586720 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.586872 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.588548 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.604671 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.606232 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:07.645153 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240502 08:04:07.665141 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:07.681339 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:07.719470 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.722147 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.722200 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.738358 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.742697 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.743029 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.743182 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:07.796833 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:07.837182 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.839278 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.839764 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.851980 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.859417 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.872642 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.881583 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:07.897387 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240502 08:04:07.932117 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:07.933403 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.934183 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:07.934963 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:07.943287 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:07.944325 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:07.967628 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:08.051403 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:08.067728 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.070686 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.072263 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.075603 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.078377 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.083683 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.110992 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:08.140255 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:08.174656 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:08.210009 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.213766 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.214489 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.215540 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.221163 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.247833 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.256536 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.272621 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.274626 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.276232 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.280669 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.288429 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.300143 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.311740 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:08.318812 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:04:08.375142 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:08.413477 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.415778 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.416174 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.416674 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.418752 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.433905 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.436359 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:08.524096 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:08.541124 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.542733 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.543519 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.544128 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.545014 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.552913 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.554256 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:08.565145 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:04:08.577136 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:08.590837 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240502 08:04:08.601428 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240502 08:04:08.639961 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:08.640020 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.649475 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.657683 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.660020 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.663764 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.680778 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.706161 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:08.733287 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:08.760032 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.760783 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.761020 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.764206 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.764967 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.771189 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.799167 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.894408 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.898587 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:08.901361 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:08.907373 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:08.909904 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:08.910058 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:08.926651 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:08.984510 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:09.020061 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.021493 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:09.022473 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:04:09.023545 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.025880 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.034489 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:09.037948 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.064853 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.229990 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.236076 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.236416 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:09.237007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.243415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:09.244500 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
I20240502 08:04:09.245329 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:09.245635 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:09.250144 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:09.258582 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:09.377713 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.384306 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.388362 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.393954 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.403683 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.407171 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.422669 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.444226 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:09.445317 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:09.450729 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.458053 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.469460 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.470914 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.477423 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:09.483613 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:09.484477 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:04:09.516496 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:09.585160 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.589409 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.593016 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.595929 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.596865 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.603071 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.618299 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:09.654181 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:09.734730 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.742272 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.743819 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.748437 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.749830 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.764626 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.768471 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:09.817019 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.819299 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.828508 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.828609 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.833457 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.848656 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.851990 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:09.874485 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:09.942694 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:09.944541 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:09.944648 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:09.945789 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.946705 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:09.953155 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:09.981384 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:10.037381 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:04:10.049890 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:10.076252 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.076912 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.078923 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.083443 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:10.085114 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.089905 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.093984 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.207839 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.209935 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.210666 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:10.213727 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.218286 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.229218 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.238700 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:10.239885 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:10.278352 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.293913 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.299422 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.300088 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.301400 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.305349 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:10.310652 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:10.316762 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:10.389691 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:10.420127 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.420943 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:10.423556 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.424176 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.424521 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.428519 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.436646 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.558465 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.558946 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.567037 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.571182 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.581468 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.583282 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.598892 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:10.630918 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:10.631784 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.632320 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.634565 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.635501 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.637360 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:10.659458 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:10.661667 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.760370 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.761067 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:10.766108 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:10.774598 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:10.774848 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.777139 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.778731 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:10.781562 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:10.882336 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:10.928077 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:04:10.962253 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:10.971177 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:10.976130 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:10.982385 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:10.991874 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:10.994550 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:11.000644 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.003814 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.005065 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:11.073491 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:04:11.084331 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:11.100867 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.101276 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:11.108702 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:11.110496 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:11.114240 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.114652 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.118039 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.123816 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:11.167476 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240502 08:04:11.230106 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.231015 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:11.233688 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.234859 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.244630 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:11.253012 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:04:11.255832 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:11.258397 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.279287 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:11.360391 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.364674 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.370153 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.374805 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:11.387671 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:11.395948 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.399163 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:11.448571 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:11.451362 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.451890 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.453506 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.455823 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:11.457345 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:11.485361 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.485780 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.589015 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.591507 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.596652 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:11.601148 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:11.607203 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.608088 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.611723 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:11.667531 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240502 08:04:11.699172 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:11.715288 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.723474 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.723888 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:11.734022 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.735718 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.759734 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:11.765089 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.808533 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.811339 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.813969 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.817032 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.837240 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:11.837280 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.837568 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:11.877624 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:11.936730 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:04:11.945502 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:11.948398 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.951560 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:11.951849 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:11.954713 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:11.955479 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.957984 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:11.978423 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.113204 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:12.119045 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.124248 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.129964 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:12.131366 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.133976 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.140655 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.144300 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.166918 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.172600 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.175886 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.179411 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.201226 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.211709 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:12.288100 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:12.301832 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.307941 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:12.309957 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.310739 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.310745 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.311017 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.318408 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.436118 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.442915 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.447985 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.449666 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.452654 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:12.457937 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.460448 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:12.490518 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:12.546964 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:12.551836 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.553733 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:12.560721 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:12.567315 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.568059 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.572876 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.579993 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.588153 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.654718 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.661538 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.663645 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.665411 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.665601 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.665867 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.696945 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:12.731755 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:04:12.760727 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240502 08:04:12.881196 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:12.896258 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.898553 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:12.898885 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:12.904665 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:12.906373 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:12.917531 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:12.928946 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:12.987650 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:13.025368 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.027040 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.027849 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.028103 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.043866 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:13.045645 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:13.046507 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:13.051704 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.154922 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.156800 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.165426 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.166065 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:13.177600 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:13.177709 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.186084 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.278752 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.286468 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.289088 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:13.295861 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:13.308140 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:13.314113 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.319643 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:13.327627 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.329353 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.355723 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.359853 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.364918 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:13.382223 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.386765 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:13.386925 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:13.400079 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.401839 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:13.449532 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:13.465746 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:13.510061 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.510319 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:13.512583 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.515640 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:13.529181 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.530968 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:13.540305 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240502 08:04:13.545311 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:13.554394 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:13.651050 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.651166 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.663282 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.669770 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.670325 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.671281 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:13.673763 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:13.676632 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:13.684438 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:13.749151 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.752355 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:13.755750 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:13.759616 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:13.778668 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:13.784548 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.794744 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.802006 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:13.828290 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:13.858407 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.860670 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:13.868276 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:13.869496 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.871080 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:13.880215 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:13.880640 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:13.889123 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:13.922098 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:14.006306 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.007037 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.010798 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.013734 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.016319 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:14.017719 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:14.052843 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:14.071075 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:04:14.128417 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.133956 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.135268 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:14.146025 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.146545 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:14.149296 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:14.149652 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:14.162240 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.200870 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:14.204638 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.208794 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.212251 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.215029 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.220171 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:14.224387 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:14.277318 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:14.346848 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.347064 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:14.348798 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:14.357695 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.358413 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.368284 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:14.371330 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:04:14.374624 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:14.380477 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:14.383401 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.488204 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.491564 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:14.494763 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.509013 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.509414 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:14.529711 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.533288 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.581362 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.585122 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:14.590341 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:14.590938 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.591876 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.594288 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.604616 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.830157 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:14.833539 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:14.838650 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:14.843632 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.855157 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.858686 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:14.863947 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.867733 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:14.884156 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:14.943553 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:14.976095 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:14.977129 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:14.983460 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:14.986532 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.001713 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.010978 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.012787 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.047111 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.055470 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.060045 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.065531 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.066443 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.069270 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.086872 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.184257 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.188278 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.189836 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.194028 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.200444 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.217998 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.220536 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:15.279177 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:15.299542 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:04:15.310765 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:15.333396 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.334022 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.345523 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.346750 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.348564 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.359294 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.370766 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.371899 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.386132 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.396168 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.396229 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.406149 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:15.420766 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:15.423914 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.425298 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.519526 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.522768 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.524308 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.525543 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.527240 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.528697 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.538975 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.657898 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.660207 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.664407 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.669759 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.670449 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.683058 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.689265 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:15.750097 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:15.796015 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.798295 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.798568 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.802116 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.816346 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:15.817281 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.828744 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:15.856580 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240502 08:04:15.860709 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:15.871690 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.872625 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:15.875066 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:15.882138 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:15.894421 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:15.897634 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:04:15.903139 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:15.919323 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:15.925908 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240502 08:04:15.948200 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:04:15.959882 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:16.008054 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.011667 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.014907 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:16.016677 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:16.020648 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:16.022497 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.024977 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.034693 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.136930 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.141453 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.144984 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:16.144965 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.161152 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:16.163429 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:16.164937 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:16.165035 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.229207 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.230381 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.238430 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.238742 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.240501 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:16.244735 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:16.252554 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:16.271230 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:16.297560 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:04:16.340422 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:16.359716 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.368484 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.369297 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.370816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.371310 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:16.385509 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:16.396842 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:16.551767 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:16.576284 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.580238 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:16.580574 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.582798 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.582794 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.587814 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:16.611294 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:16.659781 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:16.692484 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.693388 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.694326 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.697129 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.701948 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:16.718740 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:16.721735 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:16.730432 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:16.744638 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:16.808046 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:16.837455 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.837821 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.844868 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.850684 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.862862 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:16.864015 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.868645 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:16.967200 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:16.971812 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:16.974881 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.978000 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:16.985999 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.002211 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.011471 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.043103 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.050870 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.058092 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.066798 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.069916 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.071245 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.081310 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:17.132998 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:17.188828 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.191202 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:17.191568 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:17.192073 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.192878 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.193934 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.194373 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:17.201689 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:04:17.203191 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:17.330003 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:17.331251 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.336277 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.346065 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.350534 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.353268 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.372581 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.384368 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.385022 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.388511 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.398897 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.401154 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.411266 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.423574 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.428774 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.525559 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.526271 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.531627 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.532409 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.539821 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.548614 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.560665 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:17.595170 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:04:17.661979 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.669320 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.672149 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.672606 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.676687 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.681308 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.691759 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:17.696494 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:17.750416 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:04:17.751137 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:17.798172 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.799397 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.810387 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:17.813026 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.813743 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:17.824062 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.832376 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.871124 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.872857 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:17.886646 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.887413 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:17.888092 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:17.889457 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:17.890014 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:17.892604 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:17.915078 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:18.003790 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.004827 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.005333 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.006702 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.020888 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.031159 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.040963 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:18.096583 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:04:18.113986 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:18.142558 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.147492 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.155755 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.157282 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.158241 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.158984 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.166357 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.190490 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.209307 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.210775 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.220175 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.222082 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.227815 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.243549 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:18.290865 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:18.468699 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:04:18.495826 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:18.496672 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.504889 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.505071 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.513027 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.516487 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.522344 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.525938 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:18.553218 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240502 08:04:18.607698 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.608088 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.619431 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.619491 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.621661 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:18.635872 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:18.636673 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.640780 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:18.702390 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:18.703958 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:18.704373 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:18.706220 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.714860 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.721302 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.723690 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.728564 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.741933 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:18.744992 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:18.836362 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.839586 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.841307 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.844684 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:18.855464 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.865434 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.869566 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:18.895622 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:18.968227 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.971904 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:18.974824 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:18.980818 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:18.984654 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:18.984954 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:18.998080 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:19.022370 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.029098 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.030946 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.031597 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.052373 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:19.053184 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.063113 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.075649 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:19.125901 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:04:19.177350 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.178802 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.184095 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.184540 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:19.186698 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:19.198069 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:19.200673 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.209425 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:19.290176 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:19.306360 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.306365 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.307755 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.310616 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.313299 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.315738 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.320909 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:19.346040 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:04:19.364960 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:19.407310 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:19.424715 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:04:19.439738 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.442584 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.448892 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.458187 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.474781 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:19.478802 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.484508 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.491861 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.493553 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.493829 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.494732 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.497593 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.505102 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:19.518203 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.633041 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.633396 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.635115 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.637850 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:19.640156 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.653605 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.675501 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:19.681584 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:19.768482 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.769573 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.776268 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.786283 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.786370 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.787108 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.804037 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:19.894418 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:19.895035 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.895046 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:19.904996 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:19.906375 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.913656 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.917268 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:19.935619 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:19.936275 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:19.986912 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:19.988814 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:19.989251 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.001802 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.005491 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.019426 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:20.020920 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.230068 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.230366 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:20.232404 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.240700 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.246011 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.249981 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.280762 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:20.353598 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:20.358937 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.364231 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.366053 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:20.368230 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:20.369654 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.371066 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:20.390954 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.397595 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:20.461603 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240502 08:04:20.483481 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:20.495177 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:20.497498 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.498685 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:20.499074 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.501595 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.523599 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.524940 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:20.530880 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:20.627694 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:20.629093 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.630688 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:20.632010 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.642302 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.645664 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.648944 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.703179 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.713686 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.720656 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.724886 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.725811 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:20.727591 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:20.741870 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:20.745106 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:20.771096 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240502 08:04:20.822619 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:20.843855 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.850656 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:20.851488 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.852748 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:20.864372 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:20.876099 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:20.879382 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:20.933405 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:20.994767 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:20.996434 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.008293 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.012538 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.021070 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.023533 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.030874 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.049316 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.050455 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:21.050531 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:21.057446 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.067410 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.076215 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.078073 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.089560 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:21.120246 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:04:21.167384 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240502 08:04:21.174829 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.178489 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.181876 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.184674 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:21.192168 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:21.194581 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.209455 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.214040 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.307962 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.316220 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.317248 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.318213 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.327306 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.337257 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.340726 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:21.366262 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:21.418133 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:21.421901 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:21.443318 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.447297 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.449095 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.455066 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.474156 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.480224 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.487388 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.488085 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.494728 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.498266 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.501591 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.505265 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.512529 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.555653 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.653681 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.654735 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.659171 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.665966 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.669847 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.671691 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.673306 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:21.766130 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:21.790916 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.796387 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.799942 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.800590 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:21.804878 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:21.806152 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.808331 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.825947 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:21.893333 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:21.896523 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:21.898967 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.899832 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:21.900856 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:21.906919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:21.914932 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:21.916630 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:21.966660 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:22.018893 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:04:22.092370 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:22.143734 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.150153 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.152185 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.155102 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.159134 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.173884 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:22.189489 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.275141 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.282111 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.287230 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.287539 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:22.296056 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.296049 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:22.319002 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:22.325542 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.356559 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.360112 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:22.369225 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.369498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.375540 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.380806 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:22.392083 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:22.397186 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.480767 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.486977 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.494624 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.496819 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.497584 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.497853 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.507604 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:22.537707 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:04:22.618603 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.623150 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.624222 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.624629 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:22.631067 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.633483 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.640549 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.705004 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.705603 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.717557 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.719365 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:22.726274 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.736995 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.745514 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.822368 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.824959 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.827826 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.833092 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:22.859733 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 58 similar messages]
I20240502 08:04:22.863111 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.866151 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.866668 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:22.928936 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:22.954000 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:22.959249 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:22.961299 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.962597 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:22.964900 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:22.968657 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:22.968961 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:22.972342 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:04:22.992283 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:23.008158 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:23.069411 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:23.073722 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.074765 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.078047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.093719 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.094805 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.096666 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.099855 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:23.154484 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:23.161557 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.165293 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.173261 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.174804 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.180368 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:23.183140 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:23.196426 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.207504 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.297102 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.298708 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.300017 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.305262 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:23.317296 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:23.320240 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.322185 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.327039 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:23.328717 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:04:23.339722 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:23.387961 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:23.421952 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.422547 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.422694 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.430200 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.448055 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.448678 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.451663 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:23.481457 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240502 08:04:23.506863 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.509979 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.511128 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.511415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.512845 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.534407 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.537679 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:23.554759 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:23.636186 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.636343 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.639268 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.648684 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:23.651924 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.652276 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.655642 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:23.688740 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:23.757062 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.759344 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.759440 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.761806 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:23.768988 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:23.788880 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:23.789625 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:23.794693 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:23.886734 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:23.970275 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:23.971300 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:23.972275 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.973498 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:23.985713 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:23.987654 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.017204 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.044898 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:24.103600 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.104331 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.104574 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.118018 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.137588 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.140681 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.147853 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.152467 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:04:24.178140 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.181082 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.192545 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.192994 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.194077 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.214161 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.215904 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.245429 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:04:24.310468 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:24.318697 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.319386 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.321108 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.327409 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.339645 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.348649 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.348980 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.432337 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240502 08:04:24.435092 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:24.449993 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.457500 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:24.458153 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:24.458395 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.459055 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.472306 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.472826 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.475773 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.524708 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:24.534310 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.535854 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.542133 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.545498 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.545979 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.552058 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.576622 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.648592 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.649135 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.652019 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.659204 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.668536 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.671300 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.681931 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.683355 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:24.777065 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.778605 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.781491 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.781641 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.790655 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.794914 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.797165 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:24.845093 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:24.850181 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:24.901383 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.902714 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.906525 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.915468 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.931771 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.937281 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.943691 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:24.972641 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:24.975437 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:24.976295 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:24.987684 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:24.996788 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.001977 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.021988 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.105301 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:25.105798 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.110878 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.116047 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.121121 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.124900 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.137667 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.238749 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.241748 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:25.247941 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.252449 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.262910 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.270608 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.273073 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.305295 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:25.313825 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:25.320192 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.323403 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.327507 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:25.334295 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:25.338816 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.347517 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.362331 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.446658 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.446842 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.447158 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:25.448794 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.457975 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.458437 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.460790 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:25.587807 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240502 08:04:25.620141 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:04:25.642652 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:25.666239 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.669049 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.669984 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:25.677609 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.690675 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.698884 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.709075 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:25.737355 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:25.737522 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:25.783145 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:25.787811 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.790650 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.792621 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.796473 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.800999 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.819865 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:25.839116 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:04:25.866093 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:25.921762 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:25.922468 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.926370 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.936695 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:25.937391 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:25.944924 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:25.948695 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.054023 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.056242 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.056335 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.066336 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.071595 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.076289 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.077999 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.117204 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.123975 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.127353 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.130328 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.131724 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:26.136780 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:04:26.143951 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:26.144181 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240502 08:04:26.161906 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:26.173900 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:26.175839 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:26.224784 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:26.262883 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.269624 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.277460 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.280511 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.281275 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.284616 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.295395 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:26.385051 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:26.403524 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:26.405673 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.411752 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.412987 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.417933 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.424643 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.425722 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.444960 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:26.499651 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:26.509945 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.510805 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.512845 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.520629 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.524721 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:26.533228 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:26.534828 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.547202 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.596972 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.600203 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.601399 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.604633 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.605089 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.608382 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.623420 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.730994 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.731675 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.734282 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.741954 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.743319 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.757927 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.758459 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:26.804307 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:26.876655 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.882515 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:26.883708 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.884848 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.885560 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.894270 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.897635 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.940575 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:26.941277 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:26.944074 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:26.944591 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.948314 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:26.949442 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:26.951462 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:26.953938 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:26.962623 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:27.069841 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.072129 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.073520 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.074110 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.087056 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.088820 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.108191 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.213456 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.213994 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.214078 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.221055 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:27.225783 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:27.229956 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.230372 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.243615 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.295614 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.295908 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.298328 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.301939 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.306419 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.307632 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.321081 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:27.334059 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:27.484589 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:27.544266 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.545019 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.547775 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.550433 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.550441 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:27.557927 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:27.565783 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.581317 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.661967 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.662247 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.670917 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.675526 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.689339 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:27.713801 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:27.726406 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.726822 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.730615 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.730818 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.735600 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.743427 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.745445 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.759238 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.773824 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:27.782085 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:27.784520 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:04:27.865917 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240502 08:04:27.866730 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:27.880012 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.882222 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:27.883002 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:27.884155 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:27.893549 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:27.909159 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:27.909271 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:27.939677 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:04:27.954591 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240502 08:04:27.989573 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:28.007246 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.019827 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.020264 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.020404 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.031325 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.032168 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.036813 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.128623 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.128623 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.131194 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.142282 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.147398 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.149595 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.156711 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:28.172734 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:28.193629 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.194550 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.213650 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.214583 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.215323 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.218838 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.233700 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.336256 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.336241 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.337074 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.338765 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.343004 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.349835 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.364349 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:28.390213 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:04:28.408968 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:28.469758 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:28.484336 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.486862 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.497011 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.498517 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.498941 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.516566 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.517139 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.543486 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:28.548170 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:28.558388 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.566941 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.567739 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.567876 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.588841 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.591020 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.663830 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.664961 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.665472 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.667434 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.667971 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:28.676537 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:28.683422 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240502 08:04:28.685765 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.701612 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.790171 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.790848 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.792383 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.795480 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:28.800669 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:28.801445 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:28.815151 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.827677 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:28.843523 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:04:28.853037 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:28.912544 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:28.916383 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.916442 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:28.918534 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:28.926779 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:28.931747 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:28.951473 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:28.962776 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:28.999861 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:29.007184 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.008561 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:29.013762 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.021304 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.024049 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.033607 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.036098 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:29.190078 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:29.254537 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:29.255760 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.256968 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.258288 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.266098 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.292824 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.305238 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:29.355536 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:04:29.379108 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.385119 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.390900 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.391857 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.401418 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:29.407394 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:29.413035 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:29.413892 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:29.473867 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:04:29.512568 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.513026 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.522500 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.524930 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.526347 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:29.531574 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:29.536703 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.543283 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:29.611164 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:29.625164 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.629141 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.633224 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:29.634230 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.653506 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.681753 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:29.685812 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.701056 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.711391 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.720486 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:29.723510 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.732751 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:29.732950 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.733239 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:29.816406 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:29.854382 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:29.854879 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.859979 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:29.866693 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:29.873417 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:29.875731 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:29.884377 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:29.889051 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.003959 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.009910 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.011771 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.014129 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.015478 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.036692 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.047715 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.059293 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.061897 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.073441 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.081993 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.082950 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.094484 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.104558 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:30.121212 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:30.165019 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:04:30.178656 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:30.188113 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.190893 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.194605 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.196686 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.205631 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.207517 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.225955 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.316160 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.319984 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.326931 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.328644 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.329061 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.330003 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.333354 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.462229 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.464597 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.465273 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.466009 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:30.473964 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:30.481840 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.482203 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.487728 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.496922 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.507969 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:30.521255 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:30.524700 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.531601 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.533457 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.548054 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.551436 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:30.575541 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:30.648829 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240502 08:04:30.649389 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.655391 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.656941 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.663450 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.673473 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.676363 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.693821 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:30.726935 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:30.784905 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.785358 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.790174 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.797698 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.804538 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.805928 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.811232 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.898717 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.900559 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:30.901288 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:30.905937 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:30.915848 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:30.915863 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:30.916157 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:30.962734 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:04:31.000511 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240502 08:04:31.009807 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:31.053733 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:31.128981 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.133986 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.136514 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.138298 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.138542 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.150162 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.152671 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:31.194456 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:04:31.264110 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:31.265964 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.277269 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.287122 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.287585 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.289660 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.300689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.303628 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.307404 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.312079 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.335036 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.350991 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.351311 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.363296 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.375174 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:31.394939 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240502 08:04:31.397861 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:31.470942 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.471225 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.477757 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:31.482322 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:31.482589 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.489274 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.492650 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.517258 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.599855 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.608901 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.608920 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.610392 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.611464 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.614087 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.614363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.718168 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.730505 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.730962 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.733292 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.733530 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.743853 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.749917 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:31.770845 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:31.804896 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.807623 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.808259 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.812130 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:31.817955 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:31.824414 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.834326 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.850672 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.943439 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:31.945770 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.946578 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:31.949396 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:31.957953 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:31.957991 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:31.962042 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:31.994187 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:32.070396 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:32.072072 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:32.072717 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.080758 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.083484 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.087947 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.102883 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.120340 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:32.128527 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:32.131490 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.146461 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.156294 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.178303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.178354 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.183496 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.183969 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:32.192600 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:32.281301 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.283380 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.287961 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.291816 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.292300 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.295467 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.297585 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:32.345969 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:32.419562 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.419689 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.425887 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.427800 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.438570 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.444710 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.445084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.506250 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.508436 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.511531 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.513906 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.515241 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:32.523977 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:32.533576 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.537055 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.618522 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.620242 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.620982 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.622653 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.622905 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.640080 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.661693 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:32.754748 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:04:32.788012 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:32.809409 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:32.844522 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.845849 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.851977 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.855199 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.857633 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.860388 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:32.873132 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:04:32.873485 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:32.873872 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:04:32.912758 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:32.967270 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:32.967326 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:32.967980 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.970912 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:32.973510 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:32.974753 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:32.982147 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.086957 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.087905 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.095517 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:33.097746 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.099397 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.114588 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.121299 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:33.146533 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:33.199015 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:33.222482 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.231647 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.240432 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.242626 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.247207 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.247645 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.247956 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.262792 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.264967 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:33.265419 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:33.278570 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.295850 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.298162 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:33.319595 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:33.328688 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:33.332334 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:33.420728 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.422430 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.422955 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.426263 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.429025 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.433315 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:33.436674 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.553539 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.553540 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.562398 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.567955 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:33.570484 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.570973 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.596748 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:33.608049 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240502 08:04:33.660413 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.663682 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.664916 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.674181 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.675117 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:33.692725 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.692751 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:33.701277 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:04:33.735929 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:04:33.746485 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:33.760491 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.766101 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.766963 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.768667 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.781440 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:33.783769 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:33.793064 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.797852 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:33.816586 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:33.838407 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:33.891554 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.894639 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:33.894695 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:33.900224 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.900981 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:33.905815 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:33.913866 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:33.950405 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:34.032557 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.038973 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.040614 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:34.042762 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.045768 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.051872 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.057019 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:34.057293 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.067112 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.079106 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:34.087886 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:34.110231 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:34.111505 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.117919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.119464 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:34.207010 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.211398 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.213277 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:34.215163 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:34.217092 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.217132 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.249956 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:34.265187 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240502 08:04:34.278656 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:34.340870 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.346130 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.350688 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.351405 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:34.358911 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:04:34.359084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:34.368968 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.399367 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:34.447677 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:34.452167 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.454247 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.461186 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.467036 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.478925 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:34.482749 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.503047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:34.605098 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:34.691069 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.694423 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:34.694795 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.695477 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.697542 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.705895 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:34.708005 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:34.720027 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:04:34.773087 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:34.819067 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.827862 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.829731 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.830868 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.841720 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:34.849304 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:34.852445 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:34.866879 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:34.886515 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:34.898240 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:34.902846 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.907061 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.907799 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:34.909916 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:34.911094 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:34.938035 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:34.944701 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.034377 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.035384 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.039391 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.039636 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.043973 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.048326 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.049432 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.158063 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.163198 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.170979 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.174842 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.184681 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.198573 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.206094 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:35.260325 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:35.271219 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.272148 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.282776 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.287915 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.289579 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:35.296888 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:35.329751 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.332901 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.362242 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.365936 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.372021 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.373476 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.377218 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.383786 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.384410 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:35.421764 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:04:35.487041 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:35.501688 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.502547 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.503557 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:35.504391 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:35.504622 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.505697 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.510692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.513211 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.632632 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.633044 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.633311 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.635116 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.643635 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.649209 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.656507 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:35.730638 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:35.732885 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.737358 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.743757 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.749614 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.758225 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.773011 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.786665 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.803674 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.807452 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:35.812247 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.830489 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.836956 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.842734 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.848503 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:35.906970 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:35.951813 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:35.952714 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:35.957072 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.960633 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:35.962383 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:35.966087 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:35.967922 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:35.976822 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:36.064428 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:36.077955 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.083636 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.086622 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.090466 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.092718 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:36.096814 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:36.105439 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240502 08:04:36.111299 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.159194 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.161862 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.163569 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.163677 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:36.165294 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.171216 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.186618 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:36.308341 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:04:36.396260 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:36.420068 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:36.420670 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:36.420898 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.424782 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.426465 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.432610 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.434437 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.451424 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:36.468876 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:36.531672 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.534523 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.544037 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.546703 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.548646 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.565594 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.580515 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:36.597395 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:36.608549 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.611486 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.615141 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.616935 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.621320 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.633795 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.637353 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:36.688983 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:04:36.733134 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240502 08:04:36.738932 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.742990 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.745920 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.759549 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.761942 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:36.762408 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.763054 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.889005 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.889693 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.892395 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:36.897928 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.903570 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.913273 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.924824 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.936051 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.949810 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:36.954842 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:36.955400 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:36.959609 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:36.973098 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:36.976110 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:37.013149 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:37.068974 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.071873 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:37.071867 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.073262 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.073603 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:37.102324 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.111316 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:37.125102 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:37.126727 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:37.200165 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.202353 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.206372 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:37.207686 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.210738 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.217669 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.234321 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:37.317065 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.320070 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:37.321725 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.323899 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.341858 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.344257 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.344419 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:37.396397 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:37.403811 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:37.405448 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.407792 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:37.412221 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:37.412773 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.413511 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.417639 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.417904 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:37.522230 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:37.526435 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:37.527405 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.528606 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.528692 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.529014 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.529435 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.567119 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:37.660686 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:37.668569 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.669039 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.671478 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.678975 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.691251 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.712424 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:37.714037 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:37.724931 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:37.734323 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:37.747943 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.749708 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.752671 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:37.760600 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:37.787671 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.789561 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.880654 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.881140 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:37.884877 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:37.885728 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.889665 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:37.891157 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:37.896790 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:37.947335 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:04:37.972185 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240502 08:04:38.005807 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:38.081722 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.083122 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.085500 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.090422 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.091600 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.105567 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.108886 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:38.119699 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:38.197575 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.201120 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.201630 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.203998 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.219152 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.229995 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.234218 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:38.258756 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:38.326048 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.327955 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.332399 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.333408 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.336608 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.347519 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.373490 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:38.391969 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:04:38.413456 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:38.441514 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:04:38.444327 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.445842 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.454839 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.460342 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.463220 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.476747 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.477551 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.535871 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.535949 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.537235 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.542840 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.544741 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.563478 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.569264 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:38.590981 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:04:38.644726 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:38.657838 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.659986 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.660795 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.663028 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.668292 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.669528 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.679911 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:38.705806 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:38.799590 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.800156 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.805459 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.814159 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.824631 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.825568 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.838366 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:38.839846 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:38.847700 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240502 08:04:38.848256 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:38.848971 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.865247 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.875238 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.885397 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:38.888211 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:38.888554 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:38.896313 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:38.994668 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:38.995326 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:38.999577 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.004473 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:39.005183 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.008862 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.030344 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:39.061787 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:04:39.073020 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240502 08:04:39.116727 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:39.121248 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.121465 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.131879 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.132612 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.134642 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:39.135178 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.162220 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:39.186947 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:39.263167 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.265265 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.271437 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:39.277662 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.280185 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.283008 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.298063 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.307476 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.319380 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:39.319664 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.327349 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.329063 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.345041 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:39.361569 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:39.408936 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:39.440708 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:39.459728 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.460165 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.461432 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:39.466477 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:39.473585 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:04:39.474207 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.477631 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.478716 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.583191 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.584370 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.592998 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.593644 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:39.595574 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.599179 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.599485 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:39.694576 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:39.698917 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.700449 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.702582 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.705423 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.725358 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:39.726683 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:39.744994 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:39.811828 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240502 08:04:39.870594 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240502 08:04:39.903321 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:39.908398 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:39.913468 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:39.914225 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.915010 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:39.917127 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:39.924633 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:39.935587 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:39.958922 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:04:40.023229 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:40.054596 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.058633 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:40.058925 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.063544 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.070469 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.071190 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.074918 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.084240 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.088009 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.088927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.091496 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.095676 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:40.116684 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.117516 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.237200 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.237799 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.238153 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.240655 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.244660 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.248947 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.254529 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:40.280222 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:40.367146 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.371215 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.371228 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.374014 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:40.379767 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.398216 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.401877 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:40.411242 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:40.480559 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.483338 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.493140 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.494036 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.494804 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:40.512209 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:40.519400 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.534799 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.544409 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.546885 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:40.558071 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:40.564906 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.569458 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.569759 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.579936 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.598388 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:40.601737 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:40.689633 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.690382 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:40.692095 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.693894 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.696100 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.698220 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.712626 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.825233 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.831223 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.835387 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:40.839452 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:40.839596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:40.850965 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:40.851549 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:40.854653 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.856606 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.913709 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.923602 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.923624 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:40.923943 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:40.927603 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:40.944327 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:40.950186 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:41.006825 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:41.011057 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.012113 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.013875 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.016583 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.016824 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.032414 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:41.039374 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:41.041018 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.100267 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:41.132944 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.134245 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.136068 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.137336 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.139166 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.139653 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.173084 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.259266 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:41.274459 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.274614 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.275141 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.276787 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.285722 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.285964 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.291049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.320109 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240502 08:04:41.327561 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.343636 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.344841 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.345407 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.349938 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.356995 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.372006 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.385876 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:41.558372 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:41.611394 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:41.621219 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.622330 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.631225 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.631284 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.639286 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.640918 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.664166 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:41.679296 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:41.719058 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.722036 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.723161 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:41.725258 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.736056 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.736320 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.750272 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.779851 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:41.859441 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.863605 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:41.864236 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.865020 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.874686 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.876631 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.881047 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:41.887918 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:41.940197 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:41.989027 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:41.991971 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:41.997488 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:41.998276 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.006990 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.013131 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.041641 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.060760 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.062547 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.066023 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.066924 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.076375 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:42.084766 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:42.084805 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:42.097585 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.188304 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.189056 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.190143 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.194037 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:42.197206 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.209226 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.218710 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.336206 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:42.337466 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.343243 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.343744 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.353050 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:42.367148 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:42.367671 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.368001 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:42.401336 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:42.421464 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.421617 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.430040 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.432210 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.432922 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.435072 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:42.453774 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:42.464534 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:42.493988 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:42.526463 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.528324 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.528764 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.531378 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:42.539229 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.540647 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.551291 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.645979 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.646548 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.648778 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.659790 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:42.660599 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.663800 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.672001 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:42.732587 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:42.771627 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.783317 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:42.785378 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:42.790248 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.792797 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.798032 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.811906 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.814919 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.833385 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:42.843693 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:42.845700 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:42.849428 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.852105 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.867560 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.875363 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:42.875841 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:42.947980 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:42.974582 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:04:42.983119 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.984367 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:42.985390 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:42.991919 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:42.994163 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.011265 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.020346 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:43.093673 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:43.111340 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.116503 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.117040 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:43.121733 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.129335 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.137924 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.161670 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:43.176070 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:43.209676 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.212013 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.219321 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.219969 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.220908 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:43.231431 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:43.251407 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:43.371536 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:43.389973 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:43.454516 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:43.456308 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.458060 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.458547 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.467133 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:43.467139 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.486274 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:43.510952 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:43.597712 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:43.601223 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.601361 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.604584 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.605604 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.617579 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:43.631145 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:43.642518 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:43.650169 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.658721 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:43.662056 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:43.664894 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.667991 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:43.677088 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.681192 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.681558 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:43.778388 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:04:43.793920 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:43.798039 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.798866 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.799671 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.799798 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:43.803828 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.804618 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:43.830420 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:43.834479 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:43.931089 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:43.933202 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:43.939113 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.941488 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:43.951359 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:43.952399 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:43.953814 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.024775 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.025200 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.028709 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.033180 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.044106 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.053277 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.060957 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:44.078234 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240502 08:04:44.119711 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.121259 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.125448 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.130458 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.145198 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:44.146031 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:44.146250 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.149646 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:44.173353 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:44.242003 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.242530 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.247952 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.249698 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.249704 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.261139 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.263549 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.388948 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.391394 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.392289 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.395761 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.407752 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.414866 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.420068 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.423403 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.428639 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:44.430665 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:44.445624 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.459851 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.465910 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.475786 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.478152 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:44.499938 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:04:44.503336 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:44.507467 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:04:44.515635 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240502 08:04:44.551684 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:44.572036 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.572346 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.572952 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.575497 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.586442 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.592631 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.592706 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:44.691011 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:44.692409 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.692394 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.695818 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.696472 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.702037 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.703830 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.747926 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.818595 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.824627 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.824625 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.825646 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.846884 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.861167 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.877218 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.913794 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:44.914112 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:44.914844 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.915743 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:44.923125 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:44.926425 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:44.927424 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:44.941793 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:04:44.946552 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:45.044646 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:45.185835 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.188213 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:45.190914 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:45.196362 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:45.199097 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:45.201890 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:45.201961 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.203732 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.223049 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:45.234864 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:45.284586 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.289284 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.293900 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.300268 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.301702 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:45.312243 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:45.315975 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:45.391005 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240502 08:04:45.420917 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.421424 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.428572 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:45.429260 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:45.442375 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.450528 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.452533 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:45.533448 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:45.552870 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.555585 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.563977 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.564353 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:45.569921 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.569921 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:45.601771 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.615350 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:45.616031 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.621045 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.628647 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.629770 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.629889 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:45.631935 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:45.643546 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:45.739262 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:45.751500 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.756033 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.758889 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:45.766532 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.768644 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.770948 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:45.778062 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.892809 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.896399 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:45.898320 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:45.899304 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:45.910831 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:45.913584 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:45.915409 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:45.920258 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:45.994624 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.002120 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:46.003643 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.004293 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.015033 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.027302 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.039099 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:46.050460 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:46.095317 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.096645 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.097576 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.098105 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.100437 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.104199 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:46.116696 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:46.123986 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:46.202769 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:04:46.228914 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:46.229243 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.229637 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.231295 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.239249 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.239871 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:46.245872 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.250036 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:46.319319 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:46.358104 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:46.378891 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.381234 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.382982 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.387698 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.389842 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.390406 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.402714 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.411656 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:46.416404 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.424196 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.440789 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:46.445977 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.446242 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.450778 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.548597 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.552665 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.553079 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.557703 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.564617 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.568714 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:46.572585 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.676597 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.678172 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.678449 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.680716 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:46.684720 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.687435 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.702406 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:46.727965 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:46.747354 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240502 08:04:46.753674 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:46.778097 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:46.790073 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.792315 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:46.792629 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:46.798856 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.804692 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:46.805570 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:46.823635 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:46.837306 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240502 08:04:46.945966 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:47.018321 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.019138 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.019150 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.019735 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.029913 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.035081 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:47.042169 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:47.045996 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.149417 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.150254 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.161075 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.161368 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.162892 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.164384 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.191058 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.232131 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.233354 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.234757 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.247505 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.248596 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.250144 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.269276 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:47.288333 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:47.302031 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:47.359580 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.362236 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.365691 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.368402 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.369304 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.370371 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.385389 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:47.389938 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:04:47.424793 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:47.509150 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.512324 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.514540 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.523839 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.527312 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.527559 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.537739 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.573360 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.578825 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.580541 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.595820 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.600490 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.600929 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.619794 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.692600 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.693264 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.699517 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.701676 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.708954 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.712016 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.738226 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:47.787338 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:04:47.798113 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:47.833434 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.834261 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.842621 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.852051 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:47.863098 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.870409 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.870988 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:47.917511 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:47.929276 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:47.971170 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:47.973059 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:47.973160 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:47.974828 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.984635 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:47.985572 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:47.992784 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240502 08:04:47.994284 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:48.022764 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.043184 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.050981 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.055511 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.058535 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.064677 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.065016 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.070376 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.164444 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.164535 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.167058 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.170327 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.183499 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.186338 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.207209 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:48.240285 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:48.304766 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.306277 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.314594 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:48.316095 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:48.316217 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.323547 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.342543 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.358363 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.390822 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:48.391117 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:48.393474 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.396616 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.398867 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.400738 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.401000 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.408650 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.505812 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.507826 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.509976 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.519361 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.520388 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.521242 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.524159 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:48.568362 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:04:48.574203 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:04:48.678966 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:48.723204 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.724756 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:48.726996 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:48.734930 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.735570 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.744659 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.754539 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.760329 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:48.826675 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:48.835471 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:48.852681 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.853839 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.860893 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.861567 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.868518 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.881800 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.883562 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:48.981482 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:48.988610 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:48.992002 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:48.992720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:48.993157 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:49.012629 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.029587 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.099913 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.104107 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.108096 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.111560 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:49.114653 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.121659 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.150499 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:49.152823 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:49.187192 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.190426 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.191744 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:49.196415 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.197242 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.199803 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.202137 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:49.241994 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:04:49.243124 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:49.273665 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240502 08:04:49.307514 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.312275 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.315317 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.315999 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.322762 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.328264 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.341651 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:49.409454 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:49.417232 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240502 08:04:49.441960 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.456092 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.456935 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.459138 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.481480 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.493660 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.501190 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:49.516551 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.517203 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.517561 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.524293 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.526314 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.527909 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.530630 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:49.553418 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:49.630241 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.639361 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.639402 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:49.640961 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:49.645578 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.649240 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:49.649649 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.672585 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:49.730356 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:04:49.760761 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.762069 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.763394 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.774055 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.774492 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.780368 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:49.785319 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:49.794721 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:49.803320 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:49.884938 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.885527 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.887882 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.895547 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.895828 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.898957 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:49.928637 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:49.939772 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:49.971318 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.971668 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:49.975093 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:49.985574 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.989003 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:49.995996 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:49.998440 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:50.012814 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
I20240502 08:04:50.094224 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:50.098090 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.101248 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.104586 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.112308 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:50.116865 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.119766 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:50.175335 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:50.233493 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.234917 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:50.236135 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.237368 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:50.243430 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.244690 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.272650 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.295179 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:50.302259 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:50.308091 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.315271 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:50.318145 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.318495 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.333730 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.340857 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:50.378401 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240502 08:04:50.396405 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:50.481384 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:04:50.496376 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:50.573578 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:50.576644 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:50.584950 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.585271 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.588622 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.610625 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.615032 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.688091 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.689232 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:50.695709 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:50.697717 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.697762 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.700016 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.703519 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:50.776973 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:50.777154 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.781163 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.785429 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:50.788988 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:50.793717 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.797534 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.808766 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:50.876817 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:50.913493 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:50.914516 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.914948 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:50.918473 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:50.928653 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:50.929725 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:50.932719 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:50.933015 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.039167 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.041092 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.053570 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.058061 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.061044 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.066691 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.066969 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:51.103986 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.108013 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.111140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.111222 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.127501 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.149035 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.149057 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:51.181809 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:51.243273 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.247092 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:51.247195 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.248200 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.249238 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.259646 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.293169 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:51.296167 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:51.390436 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.391000 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.393481 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.403308 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:51.407164 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:51.412326 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:51.414122 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:51.423528 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.428436 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:51.453665 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:51.477566 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.492331 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.494931 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.496289 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.498006 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.498708 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.512346 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:51.528028 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:51.579324 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.579344 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.580406 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.583842 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:51.588968 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.601300 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.604929 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:51.626034 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:51.692116 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.695746 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.696913 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.697171 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.700865 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.701858 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:51.721928 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:51.816586 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:51.835180 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.838372 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.841578 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.857465 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.859968 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.862040 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.867311 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:51.891321 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:51.904209 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.906383 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:51.912375 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:51.914438 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:51.923899 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:51.940438 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:51.988654 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:52.032932 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.032932 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.038241 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:52.038816 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.046416 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.050299 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:52.052630 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:52.109185 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240502 08:04:52.115170 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:52.189788 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.192545 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.192647 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.194473 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.199855 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:52.200632 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:52.204542 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:52.231395 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:52.260903 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:52.283893 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:52.369693 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:52.372537 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.373373 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.376955 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.388599 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.401577 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.405725 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:52.511801 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:52.515499 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:52.517123 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.519163 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.520398 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:52.521770 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.523818 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:52.537087 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:52.549476 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:52.572124 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.573356 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.574242 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.574440 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.580857 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:52.589823 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:52.600422 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:52.606294 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:52.692795 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:52.696240 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.698400 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.705698 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.708442 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:52.709300 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.732779 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.734114 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:52.801620 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:52.830111 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:52.830658 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:52.834282 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.846055 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.850968 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.857437 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.858139 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.964515 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.968859 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:52.969134 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:52.972100 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.975200 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:52.976624 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:52.987506 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:53.037361 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:53.038312 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.042762 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:53.043360 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.047659 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.052057 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.070693 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:53.079595 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:53.081974 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:53.131228 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:04:53.131963 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:53.175158 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.178951 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.181200 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.183153 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:53.184446 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:53.185518 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.189826 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.201792 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.306751 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.312755 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.314188 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.321233 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.324579 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.339449 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:53.349339 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:53.349885 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.396435 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.399066 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.399179 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.399957 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.400048 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.405707 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:53.440285 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.486917 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.491585 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.493180 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.500231 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.503710 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.505388 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:53.519703 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:53.572048 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:53.577633 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:53.608747 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.610965 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.611061 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.628024 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.632475 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:53.632782 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.633039 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:53.692492 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:53.749665 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.757239 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.760906 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:53.767505 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:53.768177 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.770412 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.785836 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:53.802888 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:53.807852 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:53.812049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.813289 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:53.814564 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:53.828608 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.842981 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:53.844619 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:53.894765 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 08:04:53.913344 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:04:53.949955 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240502 08:04:54.133077 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:54.154485 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.159057 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.178881 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.182219 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.182927 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.184747 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.184955 30195 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.185006 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.186196 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.195030 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.198725 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:54.208060 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:54.210249 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.222468 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.230674 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.346534 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.348856 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.350523 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.357779 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.373914 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.376562 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.377084 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.471484 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.476922 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.480654 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.488754 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.492527 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.493170 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.494179 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:54.516541 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:04:54.555389 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:54.588363 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.588446 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.591511 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.599927 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.614367 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.618495 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:54.626164 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:54.628125 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:54.663251 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:54.674695 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.675740 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.678023 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.688081 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.688302 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.688372 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.698493 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:54.710856 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:54.748840 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:04:54.749516 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240502 08:04:54.798858 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:54.802135 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:54.802366 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:54.803720 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.807778 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.819257 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.823637 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.833839 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.955452 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.960141 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.962576 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.964207 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.971392 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:54.978976 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:54.983659 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:54.989925 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:54.997943 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.001320 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.005793 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.012815 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.017947 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:55.028831 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:55.049376 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:55.062775 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:55.127113 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.128100 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:55.129051 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.131047 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.147734 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.152711 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.154448 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:55.178490 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240502 08:04:55.179956 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:55.229799 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:55.243681 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.250115 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.251216 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.251613 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.254398 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.264146 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:55.286870 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.362345 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.366166 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.369904 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.378333 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:55.384837 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.391340 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.393144 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:55.433904 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:55.449769 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:55.449853 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.451736 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.454412 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:55.455057 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:55.456885 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.466830 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.467844 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:55.541285 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240502 08:04:55.570780 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.577315 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.577344 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.579335 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:55.581291 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.591459 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.594542 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:55.697448 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:55.703722 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.704924 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.705722 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:55.708187 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.709504 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.713654 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:55.749233 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:55.802649 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:55.851001 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:55.900686 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.904528 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.906682 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:55.912619 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:55.912918 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:55.920502 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:55.922847 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:55.923833 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.038944 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.043745 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:56.052704 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.061342 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.061831 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.063303 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.073239 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.079514 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.085412 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.095346 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.096836 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.121986 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:56.124094 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.124905 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:56.145849 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:56.226584 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.230007 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.231650 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.244305 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.246578 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.247867 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:56.250952 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:56.281265 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:56.350168 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:56.353003 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.353528 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.356766 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.362676 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.372447 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.373080 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.377647 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:56.403437 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:56.485616 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.500332 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.511160 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:56.522732 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.524663 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.526877 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.534771 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.541036 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.544061 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.546259 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.549465 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.552350 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:56.553793 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.589519 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:56.671156 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:56.683215 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:56.683324 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:56.683771 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.685752 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.692603 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.693941 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.707119 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.707741 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:56.732864 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:56.812332 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.814241 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.824072 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.830330 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:56.832610 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.833554 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.833793 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:56.912290 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:56.952003 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.957317 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.959285 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.972268 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.972672 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:56.976787 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:56.987358 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:56.992759 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:56.994078 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.010277 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.014747 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.020635 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.032815 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.048107 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.137137 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:57.138806 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:57.143337 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.144907 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.154284 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.154548 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:57.168183 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:57.185981 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:57.187585 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240502 08:04:57.234838 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:57.290966 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:57.291783 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.307180 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:57.307551 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:57.315537 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.323230 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.323626 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:57.327773 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:57.377597 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:57.416755 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.418931 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:57.422103 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:57.422574 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.426136 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:57.429279 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.429466 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:57.439903 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:57.629025 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:57.650897 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:57.656365 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:57.660509 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.662061 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.662899 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:57.673812 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.687453 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.694540 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:57.728518 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:57.787281 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.792138 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.797139 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:57.797720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.800598 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.800694 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:57.805454 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:57.871083 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:57.876155 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:57.877429 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:04:57.880882 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.882167 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:57.884093 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:57.888693 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:57.893301 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:57.907147 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:57.977792 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:04:58.013434 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.014276 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.014463 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.021088 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.031384 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.045722 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.046955 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.136579 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:58.138006 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
I20240502 08:04:58.139284 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.145326 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:04:58.161430 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:58.165789 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.169307 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.177423 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.190686 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.191004 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.193007 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.195197 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.205298 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.237223 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.238685 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.241768 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:58.281315 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:04:58.298267 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:58.341902 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.344221 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.346653 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.348742 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.359185 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.362581 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.386452 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.487614 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.490036 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.490315 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.491056 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.491567 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.494747 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.511010 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:58.544507 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:58.573187 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.574656 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.579268 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.582895 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.588955 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.598472 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.600675 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:58.654460 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:58.667403 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.668406 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.670907 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.672061 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.679348 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.696410 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.700317 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:58.734086 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:58.788971 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.790834 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.795550 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.796833 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.802520 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.809403 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.817169 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:58.905289 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:04:58.936802 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.937968 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.943081 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:58.946314 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:04:58.946820 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:58.954010 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.954363 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.956712 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:58.962451 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.963053 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:58.970077 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:58.979115 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:58.996526 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:59.000558 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:04:59.000978 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.023439 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:59.065799 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:04:59.111728 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.112175 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:59.113935 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.121939 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.126276 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:59.133358 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:59.136742 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.155229 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:59.193024 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:04:59.315179 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:04:59.320833 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.322325 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.322521 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.323590 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:59.323781 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.324743 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:59.349785 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:59.376896 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240502 08:04:59.444645 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:59.446384 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.450613 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:59.456802 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.458261 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.459368 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.466428 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.573725 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.576498 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:59.577162 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.577620 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.580111 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.586117 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:04:59.591037 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:59.607627 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:59.631408 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:04:59.641798 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240502 08:04:59.642673 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:04:59.688504 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.689667 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:59.700294 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.710912 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.733419 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.736532 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:59.744232 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.747130 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:04:59.762826 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.765383 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.774701 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.777666 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.779688 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:59.784821 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:04:59.789494 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:04:59.904214 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:04:59.904706 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:04:59.907919 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.909587 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:04:59.917131 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:04:59.919832 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:04:59.919832 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:04:59.923871 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:04:59.941236 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:05:00.032490 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.033473 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.039949 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.042888 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.056527 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.056797 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.061295 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:00.077858 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:05:00.107281 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:00.129933 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.131623 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.132180 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.133579 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.149466 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.152448 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.167582 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.219494 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.219774 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.223554 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.226936 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:00.239164 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240502 08:05:00.240134 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:00.241699 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.241816 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.242342 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.348874 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.352435 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.352736 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.352800 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.355558 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.356061 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.376652 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.493956 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.495384 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.505403 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.505966 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.510399 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.514796 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.517014 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.526044 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.538439 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.539927 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.542827 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.555104 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.559226 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:00.562038 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:05:00.565119 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240502 08:05:00.576817 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:00.597096 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:00.669346 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.669988 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.673496 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:00.675984 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:05:00.680047 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.681308 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.685531 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.716843 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:00.730408 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:00.738463 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:00.800644 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.801903 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.804785 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.806309 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.811719 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:00.822834 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.832136 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.921685 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.928143 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:00.928261 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.932439 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:00.932778 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:00.934305 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:00.954246 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:01.072201 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:05:01.133008 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.134416 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.139683 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.143668 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.145395 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.148656 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.172804 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:01.204201 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:01.252581 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.259039 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.259150 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.260185 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.281985 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.289867 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.290932 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:01.297530 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:01.300032 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:05:01.303743 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:01.325301 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.335042 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.335407 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.335920 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.340123 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:01.350884 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:01.357873 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.370486 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.456570 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.458385 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.459878 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.466347 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.473927 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.478094 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.492805 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.592031 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.595450 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.598963 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.612597 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.614851 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.621028 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.636535 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.648423 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.648960 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.663538 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.666527 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.671278 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.684754 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.686496 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:01.709908 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:01.777699 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:01.785760 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.789634 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.789875 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.796317 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.798523 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.810102 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.810824 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:01.824762 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240502 08:05:01.899123 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:01.919888 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:01.921985 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.923838 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:01.924326 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:01.932786 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:01.938766 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:01.940443 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.029929 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.034579 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.043530 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.045854 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.059513 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.062650 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.068648 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:02.115460 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.119199 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.122754 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.124871 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.130043 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.131999 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.132241 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:02.146348 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240502 08:05:02.185814 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:02.230990 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:05:02.233461 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:02.236315 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.239202 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.239784 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.241459 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:02.242327 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:02.249752 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.260532 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:02.362310 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.368000 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.368257 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.375136 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.379724 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.391638 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:02.397711 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:02.414626 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:02.460606 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.462420 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.464211 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.464721 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.466629 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.470940 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:02.486146 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.563932 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.564553 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.570503 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.574237 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.577082 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:02.579722 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.580781 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:02.682649 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240502 08:05:02.706209 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:02.727164 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:02.810010 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.813498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.814745 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.815527 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.821233 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.832495 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:02.840078 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.929502 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.930020 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:02.931674 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:02.941866 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:02.943457 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.947484 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:02.959050 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:03.025900 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:05:03.054986 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.055657 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.055886 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.059859 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.070245 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:03.078934 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.083202 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:03.090658 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:05:03.107954 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:03.147322 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240502 08:05:03.149761 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:03.187109 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.190683 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.191035 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.199641 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.203928 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.215435 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:03.225627 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:03.234793 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:03.244953 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.247690 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.252949 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.254503 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.262601 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.266657 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.291492 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:03.307313 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:03.381625 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.386551 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.387605 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.388240 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.396585 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.408921 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:03.425333 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:03.462323 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:03.522047 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.524197 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.524210 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.525859 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.525861 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.537323 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:03.547726 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:03.554819 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:03.632876 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.633662 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.639758 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.645198 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:03.645629 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:03.649922 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.674111 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.681320 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.713948 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.716956 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.717907 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.725126 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.729066 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.729157 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.745388 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:03.788790 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:03.838657 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.853049 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.853348 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.854987 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.855619 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.864054 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:03.872083 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:03.878361 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:03.973727 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:03.978869 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.979851 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:03.983342 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:03.984804 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.030306 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.031311 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.032088 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.035563 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.036331 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.036487 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:04.045533 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:04.069034 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.070900 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.084573 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:04.085070 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:05:04.136284 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:04.161021 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:05:04.161991 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.162910 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.170749 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.177284 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.180155 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.180277 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.191922 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:04.254276 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:04.283839 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.292528 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.292843 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.293102 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.293316 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.317729 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.318240 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:04.319170 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:04.402961 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.403383 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.404057 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.409394 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.417280 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.422636 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:04.426952 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:04.438134 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:04.455448 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:04.538328 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:04.601637 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240502 08:05:04.614800 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.617048 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.617412 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.621203 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.624136 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.632654 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.635406 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.746147 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.752923 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.753533 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.757120 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.769277 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.769600 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.782637 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.784790 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.798637 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.801467 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:04.801970 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:05:04.809037 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.810547 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.832823 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.834249 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:04.915481 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:05:04.919636 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240502 08:05:04.920369 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:04.944586 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:04.948647 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.949342 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:04.950012 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:04.952868 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:04.972095 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:04.975816 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:04.997846 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:05.017922 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:05.069388 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.071413 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:05.075448 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.089768 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.091406 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.097930 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.101594 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.189450 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.192562 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.193022 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:05.195549 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.203608 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.205562 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.227468 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.260843 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.270354 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.272010 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.279413 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.282478 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.284535 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.297073 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:05.395954 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:05:05.400257 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.401712 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.402384 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.403928 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:05.408145 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.418205 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:05:05.421289 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:05.427711 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:05.455360 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:05:05.464069 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:05.533555 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.533759 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.534257 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.539395 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.539579 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:05.544384 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.550729 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:05.585686 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:05.609119 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.616156 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.619267 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.624711 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.631031 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:05.633983 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.638723 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:05.673502 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:05.712239 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.714728 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.719107 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.720726 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.721159 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.722220 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.742336 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:05.791738 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:05.836747 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.845347 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.848395 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:05.849344 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:05.859263 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:05.867209 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:05.872651 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.880614 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:05.916237 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240502 08:05:05.967931 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.977468 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:05.977720 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:05.997516 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:06.010399 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.014101 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.030689 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.032281 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.033015 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.033608 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.064558 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:06.065366 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.066217 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.083070 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:06.155094 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:06.295354 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:06.351979 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.354735 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.362417 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.362879 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.364970 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:06.375592 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.378850 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:06.398034 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:06.414654 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:06.439340 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.441875 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.447242 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.447937 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.454595 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.455220 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.460141 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:06.499792 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:06.563539 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.563997 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:06.564324 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.566720 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.571206 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.572762 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.579834 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:06.587996 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:06.705603 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:06.709354 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.715093 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.717803 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.721581 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.725143 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.729817 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.729974 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.743832 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:06.762176 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:06.768775 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.773259 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.777029 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:06.786193 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.800423 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:06.813545 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:06.899729 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.901482 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:06.904152 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:06.907897 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.911944 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:06.926822 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:06.937125 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:06.948946 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:06.954505 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:05:07.028419 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240502 08:05:07.030759 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.036056 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.044118 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.045934 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.047226 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.047425 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.056980 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.126679 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.129783 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.132009 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.134328 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.152138 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.171054 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.174548 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:07.208626 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:07.221490 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.225701 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.225701 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.233723 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.244956 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.249076 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.258389 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:07.317561 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:07.347522 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.348021 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.350999 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.359462 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.363610 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.386073 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.398311 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:07.468691 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:07.480850 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.486503 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.496687 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.516357 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.524577 28509 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.524932 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.534983 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.535763 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.544139 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.549552 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.552196 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:07.559749 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:07.567239 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.577883 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.582382 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:07.637751 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:07.679544 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.679819 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.680629 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.682299 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.689083 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.689343 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.689791 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.807260 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.808840 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.812264 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.812381 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.818305 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.831728 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.836609 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:07.919680 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:07.926424 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:07.928476 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:07.931236 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.938668 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:07.946900 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:07.962586 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:07.963975 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:08.012696 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:05:08.053620 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:08.146795 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.147240 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.148170 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:08.154808 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240502 08:05:08.157646 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.159528 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:08.165047 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:08.178543 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:08.179618 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:08.200853 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:08.280622 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.284294 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.284693 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.288177 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.298867 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:08.302938 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.312981 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:08.328579 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:08.336370 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.340137 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.342788 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.352131 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:08.364495 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.374198 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.380661 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:08.441174 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:08.473265 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.473671 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.490531 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.491046 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:08.492849 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:08.500628 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.511449 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:08.542786 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:08.603283 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.603302 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.608657 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.609930 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:08.617801 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:08.618145 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.621310 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.623688 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:08.722445 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.723169 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.731510 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.732421 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.750995 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:08.751096 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:08.777208 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:08.801769 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:08.807770 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.811044 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:08.815272 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.817621 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.819655 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:08.826485 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.830044 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:08.944074 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:08.946106 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:08.946211 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:08.953861 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:08.956044 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:08.956631 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:08.957824 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:08.968183 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:08.973495 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:08.973692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:08.976662 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:09.092398 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:09.094118 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:09.094801 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.096107 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.105865 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.111617 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.118006 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.126996 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.129242 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.132766 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:09.147526 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.148640 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:09.167768 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.169471 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:09.180514 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:09.194023 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:09.271237 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.273702 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:09.276852 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240502 08:05:09.283679 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.283993 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.286378 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.297824 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:09.306903 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:09.392725 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240502 08:05:09.401731 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:09.408988 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.409664 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:09.411343 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:09.417667 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:09.420650 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:09.425022 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.428222 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.437791 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.548405 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:09.551502 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.553642 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.556350 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:09.573797 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.574075 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.575453 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:09.579294 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.587692 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.590751 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.591054 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:09.621161 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.625528 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.636672 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:09.747272 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:09.823807 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:05:09.860576 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240502 08:05:09.927282 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240502 08:05:09.942476 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.947518 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.954090 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:09.957355 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:09.957453 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.971762 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:09.975893 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:09.981304 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:05:09.982661 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:09.997368 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:09.999868 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.004426 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.004913 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.035001 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.035562 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.037371 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:10.144799 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.145303 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.145891 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.149389 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.155962 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:10.174862 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.193542 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:10.262080 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:10.284126 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.284472 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.290956 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.292706 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.304351 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:10.305752 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.316108 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.405023 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.405086 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.406234 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.409040 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.418498 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.427295 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:10.427640 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240502 08:05:10.448288 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:10.478797 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:10.495656 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.498907 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.502410 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:10.503548 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.509804 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.521822 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.524181 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:10.529114 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:10.626415 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.627810 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.628278 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.630311 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.632552 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.639209 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.666003 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:10.693727 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:05:10.736264 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240502 08:05:10.749145 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:10.769953 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.780598 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.786621 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.798418 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.809309 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.810967 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.816792 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.820164 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.821432 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.821980 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:10.828253 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:10.837206 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.845091 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:10.873924 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:10.891938 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:10.975482 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.978471 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.978834 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:10.981434 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:10.983194 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:10.995101 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:11.003868 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:11.012594 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:11.067425 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:11.095155 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.099692 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:11.101836 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.102670 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.112720 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.119627 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.122203 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:11.186830 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:11.225888 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.227959 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:11.228396 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:11.229135 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.229924 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.231796 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:11.240538 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.260828 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.304378 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.305172 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.308197 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:11.308601 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.312647 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.322062 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:11.345768 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:11.350240 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:05:11.373121 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:11.440965 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.441228 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.443794 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.444478 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.446873 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.455475 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:11.464640 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:11.516800 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:05:11.535677 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:11.582932 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.584877 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.585640 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.589679 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:11.600378 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:11.603580 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.604671 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.771343 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.778786 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.779270 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:11.780648 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.785766 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:11.793951 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:11.795271 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:11.807734 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:11.839550 28463 consensus_peers.cc:485] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:11.909673 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.909680 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:11.910948 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:11.915496 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:11.928982 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.932961 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:11.933822 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:11.969244 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:11.992004 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:11.996217 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.000826 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.001123 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.009097 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.017155 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:12.020004 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:12.045233 28725 consensus_peers.cc:485] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:12.102576 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.106353 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.111235 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.111362 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:12.119364 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240502 08:05:12.120015 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.122817 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.131695 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:12.177850 28725 consensus_peers.cc:485] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:12.226426 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.228806 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.229537 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.232092 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.243988 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:12.244876 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.256534 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:12.278782 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:12.317353 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:12.346949 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.349431 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.354735 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.365300 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.366223 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.371035 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:12.380096 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.431684 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.437498 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.441416 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.441828 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.442066 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.444025 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:12.461230 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:12.505656 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:12.572633 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.579133 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.579510 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.579795 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:12.580842 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.596916 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.603798 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.709440 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.711009 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.727191 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:12.729791 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:12.731302 30326 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.731316 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.732858 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.740101 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.742827 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.749920 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.754599 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.780568 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:12.784008 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.787868 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.791906 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:12.894959 28463 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:12.900281 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:12.900573 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.911978 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.912498 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:12.914551 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:12.926470 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:12.932008 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:12.941475 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:13.023495 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.031675 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.033653 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.037310 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.037520 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:13.042541 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.055084 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:13.057286 28463 consensus_peers.cc:462] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:13.147233 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.152598 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.155683 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.156677 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.167258 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
W20240502 08:05:13.181744 28463 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240502 08:05:13.182598 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:13.205597 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:13.208402 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.222895 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.223641 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.234534 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.241078 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:13.241349 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:13.253679 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:13.284325 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:13.448269 28463 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240502 08:05:13.498610 28594 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:13.543874 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:13.548560 28594 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:13.549719 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.553493 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:13.553709 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:13.569258 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.569974 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:13.571166 28463 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240502 08:05:13.573199 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.610918 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.625962 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.632349 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.632401 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:13.640627 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.642087 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:13.652799 13881 raft_consensus.cc:998] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.170s)
I20240502 08:05:13.653252 13881 consensus_queue.cc:237] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:13.657347 28790 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:13.658841 30477 raft_consensus.cc:1261] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:13.659840 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.659847 13881 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:13.660113 13881 consensus_queue.cc:1035] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:13.664659 13881 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:13.667327 28790 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:13.670009 28256 catalog_manager.cc:5484] T a7fd7f8583e14c91a932767ebe23b419 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:13.672745 30477 raft_consensus.cc:2932] T a7fd7f8583e14c91a932767ebe23b419 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
W20240502 08:05:13.681340 28241 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:13.682298 28241 catalog_manager.cc:4573] TS e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet a7fd7f8583e14c91a932767ebe23b419: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:13.763609 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.765482 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.768769 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.770306 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:13.774956 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.775797 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:13.794044 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:13.885380 28594 consensus_peers.cc:462] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:13.915630 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.915827 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.915843 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.926172 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.927603 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:13.934088 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 08:05:13.939963 28725 consensus_peers.cc:485] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:13.948065 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:13.954372 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.955566 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 08:05:13.955929 30148 consensus_peers.cc:485] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:13.957170 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:13.957684 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:13.971786 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:13.981022 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:13.990084 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.007288 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.043068 14697 raft_consensus.cc:998] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.225s)
I20240502 08:05:14.043491 14697 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.045053 28397 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:14.045331 30346 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:14.045776 14694 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:14.045970 14692 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:14.047977 14694 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.048848 30346 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.051541 28253 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.052857 28397 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.087653 14626 raft_consensus.cc:998] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.080s)
I20240502 08:05:14.088193 14626 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.090590 28397 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:14.090613 30477 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:14.091377 30346 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:14.091681 14626 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:14.091866 14879 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:14.092237 14879 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:14.097699 14878 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.100840 30477 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.101101 30346 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.103509 28397 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.104341 28253 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.115445 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.120842 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.121311 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.126838 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.135619 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.136236 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.144840 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.145011 14692 raft_consensus.cc:998] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.086s)
I20240502 08:05:14.145390 14692 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:14.149006 30477 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:14.149551 14697 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:14.161070 14697 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.163810 30477 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.165993 28252 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.174084 28529 consensus_queue.cc:237] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.175671 30477 raft_consensus.cc:1261] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:05:14.176052 28465 consensus_peers.cc:485] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6efe17dcca7b4babad4785d19552b699 (127.16.36.198:45421): Couldn't send request to peer 6efe17dcca7b4babad4785d19552b699. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7dbf7921ae84476ab53fbd4707f463eb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:14.176255 14692 consensus_queue.cc:1035] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:14.177981 14694 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.178431 30477 raft_consensus.cc:2932] T 7dbf7921ae84476ab53fbd4707f463eb P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.180198 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 7dbf7921ae84476ab53fbd4707f463eb with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:14.180928 28254 catalog_manager.cc:5484] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 8 to 9, VOTER 6efe17dcca7b4babad4785d19552b699 (127.16.36.198) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.224370 14694 raft_consensus.cc:998] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.323s)
I20240502 08:05:14.224795 14694 consensus_queue.cc:237] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.226217 28660 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.226675 14694 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.226773 30346 raft_consensus.cc:1261] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.227260 14694 consensus_queue.cc:1035] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.228638 14694 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.230171 28660 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.230971 30346 raft_consensus.cc:2932] T 07461c988fb3402c91cfd9235efbb9b1 P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.231141 28254 catalog_manager.cc:5484] T 07461c988fb3402c91cfd9235efbb9b1 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.265329 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.267452 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.270812 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.273514 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.282482 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.299010 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.318607 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.365846 14878 raft_consensus.cc:998] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.010s)
I20240502 08:05:14.366308 14878 consensus_queue.cc:237] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:05:14.367741 30477 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:14.367748 28529 raft_consensus.cc:1261] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:14.368187 14879 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:14.368533 14879 consensus_queue.cc:1035] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:14.369395 14672 raft_consensus.cc:998] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.425s)
I20240502 08:05:14.369803 14672 consensus_queue.cc:237] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:14.371429 28398 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.371933 30476 raft_consensus.cc:1261] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.372399 14672 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.372686 14672 raft_consensus.cc:998] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.257s)
I20240502 08:05:14.373001 14672 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:14.373448 14991 consensus_queue.cc:1035] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.374293 30476 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:14.375777 14994 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:14.376180 14672 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.377781 30476 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.380569 28256 catalog_manager.cc:5484] T d08ad2a981fc4c13bc89d22feea7d556 P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.380831 28398 raft_consensus.cc:2932] T d08ad2a981fc4c13bc89d22feea7d556 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.383301 30475 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.385565 14672 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.386420 14692 raft_consensus.cc:998] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.375s)
I20240502 08:05:14.386775 14692 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:14.388015 28790 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.389060 30475 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.389235 28256 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.389453 14697 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.391366 14697 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.398264 28660 consensus_queue.cc:237] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:05:14.401357 14876 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:05:14.402784 14694 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.403245 30477 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.405298 30475 raft_consensus.cc:1261] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:14.406121 28790 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.406519 28252 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.407425 28529 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:05:14.407543 30475 raft_consensus.cc:2932] T 6a40382745f84f0eadf9e0ee2e057289 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:05:14.409888 28254 catalog_manager.cc:5484] T 6a40382745f84f0eadf9e0ee2e057289 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 8 to 9, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
W20240502 08:05:14.412570 28594 consensus_peers.cc:485] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581): Couldn't send request to peer c39bcb719db947b8afb0bb141e1f7a4e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f98044a1730e4a04a7088199d4f0850e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:14.413385 15000 consensus_queue.cc:1035] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:14.416877 15000 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:05:14.417490 30475 raft_consensus.cc:2932] T f98044a1730e4a04a7088199d4f0850e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:05:14.418174 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.420096 28243 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet f98044a1730e4a04a7088199d4f0850e with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:14.422544 28251 catalog_manager.cc:5484] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 8 to 9, VOTER c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194) added. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } }
I20240502 08:05:14.424057 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.428409 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.436810 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.439579 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.458530 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.469278 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.485987 14626 raft_consensus.cc:998] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.203s)
I20240502 08:05:14.486357 14626 consensus_queue.cc:237] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.490131 30346 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:14.490952 14876 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:14.492123 30213 raft_consensus.cc:1261] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:14.493561 14876 consensus_queue.cc:1035] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:14.497421 14876 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.499914 30213 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6c68c916f39a45f697df7eaf8973225a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.500176 28256 catalog_manager.cc:5484] T 0499a3f3e6bd4069bf992e32f1abfdb0 P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 2 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.504623 30346 raft_consensus.cc:2932] T 0499a3f3e6bd4069bf992e32f1abfdb0 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.521593 30657 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:58.290100 30665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:58.290388 30665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:58.296689 30665 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:59.179301 30665 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:59.814186 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.814347 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.814430 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.814483 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.814577 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.814644 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.814718 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.814786 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.814824 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.814863 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.814903 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.814952 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.814993 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.815045 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:00.093653 30665 rebalancer_tool.cc:1074] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:00.150804 30665 rebalancer_tool.cc:1074] tablet 7dbf7921ae84476ab53fbd4707f463eb: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:00.232406 30665 rebalancer_tool.cc:1074] tablet 07461c988fb3402c91cfd9235efbb9b1: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:00.309939 30665 rebalancer_tool.cc:1074] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.444497 30665 rebalancer_tool.cc:1074] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:00.495733 30665 rebalancer_tool.cc:1074] tablet c47d608895f94a2c82e695ca16677abf: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:00.565304 30665 rebalancer_tool.cc:1074] tablet 3288b438832e488e8257227cb8934565: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:00.720307 30665 rebalancer_tool.cc:1074] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.967895 30665 rebalancer_tool.cc:1074] tablet 1631d3e38209497ea2454fdfc555a319: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:01.242373 30665 rebalancer_tool.cc:1074] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:01.410410 30665 rebalancer_tool.cc:1074] tablet 2ba62cb7ab7b429d85e0ac037c127cb4: '71922d97e1a4414698ecd14a7b2dedec' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:01.678983 30665 rebalancer_tool.cc:1074] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:01.831300 30665 rebalancer_tool.cc:1074] tablet 4cb29b8b566747e5b82a8969a070123e: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:01.849460 30665 rebalancer_tool.cc:1084] tablet ab785e07fa8248e5889bea8ac755d002: 'de423605477a4819a9569821f59f03e4' -> '6efe17dcca7b4babad4785d19552b699' move ignored: Illegal state: tablet ab785e07fa8248e5889bea8ac755d002: replica 6efe17dcca7b4babad4785d19552b699 is already present
I20240502 08:00:01.849570 30665 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:02.406807 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.406916 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.406992 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.407086 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.407179 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.407223 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.407281 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.407332 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.407377 30665 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
I20240502 08:00:02.546757 30665 rebalancer_tool.cc:1074] tablet 57f7c358ace742649df52380939ece9d: 'e5e9553dabc94f218f6427c1685ffc38' -> '' move scheduled
I20240502 08:00:02.586592 30665 rebalancer_tool.cc:1074] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:02.609105 30665 rebalancer_tool.cc:1074] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:02.632359 30665 rebalancer_tool.cc:1074] tablet c65fe7f823df4b4996769fc784d3adb7: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:02.642288 30665 rebalancer_tool.cc:1074] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:08.582062 30665 rebalancer_tool.cc:1136] tablet 7dbf7921ae84476ab53fbd4707f463eb: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:10.167423 30665 rebalancer_tool.cc:1136] tablet 57f7c358ace742649df52380939ece9d: 'e5e9553dabc94f218f6427c1685ffc38' -> '' move completed: OK
I20240502 08:00:10.809886 30665 rebalancer_tool.cc:1123] tablet 1631d3e38209497ea2454fdfc555a319: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move is abandoned: Illegal state: Leader has not yet committed an operation in its own term
I20240502 08:00:12.613724 30665 rebalancer_tool.cc:1136] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:12.630101 30665 rebalancer_tool.cc:1136] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:12.648074 30665 rebalancer_tool.cc:1136] tablet 4cb29b8b566747e5b82a8969a070123e: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:12.663075 30665 rebalancer_tool.cc:1074] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:12.795776 30665 rebalancer_tool.cc:1074] tablet 698f494a22d34e659766b97cb24d0293: '71922d97e1a4414698ecd14a7b2dedec' -> '' move scheduled
I20240502 08:00:12.806022 30665 rebalancer_tool.cc:1136] tablet 698f494a22d34e659766b97cb24d0293: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:12.820749 30665 rebalancer_tool.cc:1136] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:12.853559 30665 rebalancer_tool.cc:1136] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:12.868244 30665 rebalancer_tool.cc:1136] tablet c65fe7f823df4b4996769fc784d3adb7: '71922d97e1a4414698ecd14a7b2dedec' -> '' move completed: OK
I20240502 08:00:12.978256 30665 rebalancer_tool.cc:1136] tablet c47d608895f94a2c82e695ca16677abf: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:13.082111 30665 rebalancer_tool.cc:1136] tablet afc9c4d9dd5b4c039686ccd918616859: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:13.095755 30665 rebalancer_tool.cc:1136] tablet 07461c988fb3402c91cfd9235efbb9b1: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:13.109833 30665 rebalancer_tool.cc:1136] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:13.122406 30665 rebalancer_tool.cc:1136] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:14.036736 30665 rebalancer_tool.cc:1123] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6c68c916f39a45f697df7eaf8973225a' move is abandoned: Not found: Server with UUID ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not a member of the config. RaftConfig: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
W20240502 08:00:14.667331 6742 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 08:00:14.405870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.36.200:43189 (local address 127.0.0.1:33744)
0502 08:00:14.405980 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0502 08:00:14.405995 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0502 08:00:14.406070 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 08:00:14.666563 (+260493us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":28}
I20240502 08:00:14.744539 30665 rebalancer_tool.cc:1136] tablet 5d1c1fa046b741aa8cf296aa57847fe3: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:15.081324 30665 rebalancer_tool.cc:1074] tablet a28d406b43194bc6b775707d021618f0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:15.089263 30665 rebalancer_tool.cc:1074] tablet 3598dbda342f49aea2384b5beb17d7a1: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:15.097520 30665 rebalancer_tool.cc:1136] tablet 3598dbda342f49aea2384b5beb17d7a1: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:15.105126 30665 rebalancer_tool.cc:1136] tablet a28d406b43194bc6b775707d021618f0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:15.334028 30665 rebalancer_tool.cc:1136] tablet 2ba62cb7ab7b429d85e0ac037c127cb4: '71922d97e1a4414698ecd14a7b2dedec' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:15.348157 30665 rebalancer_tool.cc:1136] tablet 04d358245cdf4b2ebb0b5005f75cf1e0: 'de423605477a4819a9569821f59f03e4' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:05:14.252713 30665 rebalancer_tool.cc:1136] tablet 3288b438832e488e8257227cb8934565: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: Incomplete: tablet 3288b438832e488e8257227cb8934565, TS 71922d97e1a4414698ecd14a7b2dedec -> TS e5e9553dabc94f218f6427c1685ffc38 move failed, target replica disappeared
Illegal state: tablet server e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): unacceptable health status UNAVAILABLE
I20240502 08:05:14.535503 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.537050 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.540071 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.542955 15000 raft_consensus.cc:998] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.276s)
I20240502 08:05:14.543319 15000 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:14.544584 28398 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.544973 15000 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.545990 30475 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:14.546545 15000 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:14.548983 14177 raft_consensus.cc:998] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.245s)
I20240502 08:05:14.549401 14177 consensus_queue.cc:237] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:14.552618 30346 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 08:05:14.553043 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.553181 28398 raft_consensus.cc:1261] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Refusing update from remote peer 6c68c916f39a45f697df7eaf8973225a: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240502 08:05:14.553555 14177 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:05:14.553829 14177 consensus_queue.cc:1035] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 08:05:14.553964 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.554594 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.554841 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.570111 15000 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.573029 30475 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.573501 28398 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:14.573748 28251 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.575882 14177 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6c68c916f39a45f697df7eaf8973225a [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:14.577984 28398 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.578444 28256 catalog_manager.cc:5484] T 5d1c1fa046b741aa8cf296aa57847fe3 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 12 to 13, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 3 leader_uuid: "6c68c916f39a45f697df7eaf8973225a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
W20240502 08:05:14.580001 28725 consensus_peers.cc:485] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240502 08:05:14.583356 30346 raft_consensus.cc:2932] T 5d1c1fa046b741aa8cf296aa57847fe3 P 6efe17dcca7b4babad4785d19552b699 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 12 to 13, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:14.584812 14838 raft_consensus.cc:998] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.308s)
I20240502 08:05:14.585131 14838 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:05:14.586231 28660 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:14.586666 14838 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:05:14.587109 28332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20240502 08:05:14.588914 15065 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:05:14.589622 28660 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:05:14.593715 28256 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 reported cstate change: config changed from index 7 to 8, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:05:14.601568 28398 consensus_queue.cc:237] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:14.603019 28660 raft_consensus.cc:1261] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 08:05:14.603209 28332 consensus_peers.cc:485] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec -> Peer 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589): Couldn't send request to peer 29126d5aee924f81bdd47cc7153420a1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3598dbda342f49aea2384b5beb17d7a1. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:14.603439 15070 consensus_queue.cc:1035] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:14.605145 15065 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.605505 28661 raft_consensus.cc:2932] T 3598dbda342f49aea2384b5beb17d7a1 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.607774 28242 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3598dbda342f49aea2384b5beb17d7a1 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:14.608420 28256 catalog_manager.cc:5484] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 8 to 9, VOTER 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199) added. New cstate: current_term: 1 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:14.679754 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.682942 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.684119 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.686034 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.686528 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.694641 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.697511 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.702868 15109 ts_tablet_manager.cc:902] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:05:14.703524 15109 tablet_copy_client.cc:308] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
W20240502 08:05:14.703598 28241 catalog_manager.cc:4573] TS e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3598dbda342f49aea2384b5beb17d7a1: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:14.704087 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 7dbf7921ae84476ab53fbd4707f463eb from peer 6efe17dcca7b4babad4785d19552b699 ({username='slave'} at 127.16.36.198:43585)
I20240502 08:05:14.704178 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 7dbf7921ae84476ab53fbd4707f463eb from peer 6efe17dcca7b4babad4785d19552b699 at {username='slave'} at 127.16.36.198:43585: session id = 6efe17dcca7b4babad4785d19552b699-7dbf7921ae84476ab53fbd4707f463eb
I20240502 08:05:14.705682 28548 tablet_copy_source_session.cc:205] T 7dbf7921ae84476ab53fbd4707f463eb P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:05:14.713400 15109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7dbf7921ae84476ab53fbd4707f463eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:14.716061 15109 tablet_copy_client.cc:783] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 5 data blocks...
I20240502 08:05:14.720857 15109 tablet_copy_client.cc:647] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:14.722100 15109 tablet_copy_client.cc:523] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:14.723714 15109 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: Bootstrap starting.
I20240502 08:05:14.735808 15109 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:14.736160 15109 tablet_bootstrap.cc:492] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: Bootstrap complete.
I20240502 08:05:14.736364 15109 ts_tablet_manager.cc:1367] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240502 08:05:14.736879 15109 raft_consensus.cc:348] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.736990 15109 raft_consensus.cc:729] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efe17dcca7b4babad4785d19552b699, State: Initialized, Role: FOLLOWER
I20240502 08:05:14.737119 15109 consensus_queue.cc:260] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.737430 15109 ts_tablet_manager.cc:1398] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:05:14.737884 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 6efe17dcca7b4babad4785d19552b699-7dbf7921ae84476ab53fbd4707f463eb received from {username='slave'} at 127.16.36.198:43585
I20240502 08:05:14.737957 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 6efe17dcca7b4babad4785d19552b699-7dbf7921ae84476ab53fbd4707f463eb on tablet 7dbf7921ae84476ab53fbd4707f463eb with peer 6efe17dcca7b4babad4785d19552b699
W20240502 08:05:14.791605 28463 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): Couldn't send request to peer e5e9553dabc94f218f6427c1685ffc38. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240502 08:05:14.807169 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.807849 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.814579 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.814903 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:14.829588 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.830729 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.845598 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:14.894883 28529 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:14.895959 30346 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:14.896314 14694 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:14.897814 14697 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.898191 30346 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.900527 28249 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 8 to 9, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:14.903616 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 3288b438832e488e8257227cb8934565 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39806
I20240502 08:05:14.904150 15177 tablet_replica.cc:330] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:05:14.904337 15177 raft_consensus.cc:2229] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:14.904538 15177 raft_consensus.cc:2258] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:14.904791 28529 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:14.905380 15177 ts_tablet_manager.cc:1875] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 08:05:14.906338 28464 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:14.907043 30346 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:14.907428 14694 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:14.909060 15177 ts_tablet_manager.cc:1888] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 08:05:14.909147 15177 log.cc:1196] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/3288b438832e488e8257227cb8934565
I20240502 08:05:14.909297 14694 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:14.909722 30346 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:14.909873 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 3288b438832e488e8257227cb8934565 (table rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]) successfully deleted
I20240502 08:05:14.911010 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3288b438832e488e8257227cb8934565 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:14.912142 28249 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 9 to 10, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:14.979851 30393 maintenance_manager.cc:419] P 6efe17dcca7b4babad4785d19552b699: Scheduling UndoDeltaBlockGCOp(7dbf7921ae84476ab53fbd4707f463eb): 4584 bytes on disk
I20240502 08:05:14.980413 30283 maintenance_manager.cc:643] P 6efe17dcca7b4babad4785d19552b699: UndoDeltaBlockGCOp(7dbf7921ae84476ab53fbd4707f463eb) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240502 08:05:14.981962 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:14.984211 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.985227 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:14.985857 15229 ts_tablet_manager.cc:902] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: Initiating tablet copy from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991)
I20240502 08:05:14.986445 15229 tablet_copy_client.cc:308] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:33991
I20240502 08:05:14.986912 28681 tablet_copy_service.cc:140] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Received BeginTabletCopySession request for tablet f98044a1730e4a04a7088199d4f0850e from peer c39bcb719db947b8afb0bb141e1f7a4e ({username='slave'} at 127.16.36.194:37059)
I20240502 08:05:14.987025 28681 tablet_copy_service.cc:161] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Beginning new tablet copy session on tablet f98044a1730e4a04a7088199d4f0850e from peer c39bcb719db947b8afb0bb141e1f7a4e at {username='slave'} at 127.16.36.194:37059: session id = c39bcb719db947b8afb0bb141e1f7a4e-f98044a1730e4a04a7088199d4f0850e
I20240502 08:05:14.988377 28681 tablet_copy_source_session.cc:205] T f98044a1730e4a04a7088199d4f0850e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:05:14.989471 15229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f98044a1730e4a04a7088199d4f0850e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:14.993702 15229 tablet_copy_client.cc:783] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 5 data blocks...
I20240502 08:05:14.994490 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:14.998909 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240502 08:05:15.004201 28725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 36 similar messages]
I20240502 08:05:15.004839 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.009692 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.030436 15229 tablet_copy_client.cc:647] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:15.031817 15229 tablet_copy_client.cc:523] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:15.033512 15229 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap starting.
I20240502 08:05:15.043969 15229 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:15.044255 15229 tablet_bootstrap.cc:492] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: Bootstrap complete.
I20240502 08:05:15.044502 15229 ts_tablet_manager.cc:1367] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 08:05:15.044879 15229 raft_consensus.cc:348] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:05:15.044975 15229 raft_consensus.cc:729] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39bcb719db947b8afb0bb141e1f7a4e, State: Initialized, Role: FOLLOWER
I20240502 08:05:15.045089 15229 consensus_queue.cc:260] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } }
I20240502 08:05:15.045339 15229 ts_tablet_manager.cc:1398] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:15.045766 28681 tablet_copy_service.cc:342] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Request end of tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-f98044a1730e4a04a7088199d4f0850e received from {username='slave'} at 127.16.36.194:37059
I20240502 08:05:15.045846 28681 tablet_copy_service.cc:433] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: ending tablet copy session c39bcb719db947b8afb0bb141e1f7a4e-f98044a1730e4a04a7088199d4f0850e on tablet f98044a1730e4a04a7088199d4f0850e with peer c39bcb719db947b8afb0bb141e1f7a4e
I20240502 08:05:15.123528 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.123673 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:15.128872 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:15.134188 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.136648 30345 raft_consensus.cc:1203] T 7dbf7921ae84476ab53fbd4707f463eb P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:05:15.142552 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.142597 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.150382 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.189100 15282 ts_tablet_manager.cc:902] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: Initiating tablet copy from peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537)
I20240502 08:05:15.189635 15282 tablet_copy_client.cc:308] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:33537
I20240502 08:05:15.194334 28419 tablet_copy_service.cc:140] P 71922d97e1a4414698ecd14a7b2dedec: Received BeginTabletCopySession request for tablet 3598dbda342f49aea2384b5beb17d7a1 from peer 29126d5aee924f81bdd47cc7153420a1 ({username='slave'} at 127.16.36.199:52255)
I20240502 08:05:15.194442 28419 tablet_copy_service.cc:161] P 71922d97e1a4414698ecd14a7b2dedec: Beginning new tablet copy session on tablet 3598dbda342f49aea2384b5beb17d7a1 from peer 29126d5aee924f81bdd47cc7153420a1 at {username='slave'} at 127.16.36.199:52255: session id = 29126d5aee924f81bdd47cc7153420a1-3598dbda342f49aea2384b5beb17d7a1
I20240502 08:05:15.195843 28419 tablet_copy_source_session.cc:205] T 3598dbda342f49aea2384b5beb17d7a1 P 71922d97e1a4414698ecd14a7b2dedec: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:05:15.197221 15282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3598dbda342f49aea2384b5beb17d7a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:15.199543 15282 tablet_copy_client.cc:783] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 5 data blocks...
I20240502 08:05:15.201469 28577 maintenance_manager.cc:419] P c39bcb719db947b8afb0bb141e1f7a4e: Scheduling UndoDeltaBlockGCOp(f98044a1730e4a04a7088199d4f0850e): 4840 bytes on disk
I20240502 08:05:15.201922 28467 maintenance_manager.cc:643] P c39bcb719db947b8afb0bb141e1f7a4e: UndoDeltaBlockGCOp(f98044a1730e4a04a7088199d4f0850e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240502 08:05:15.204716 28790 consensus_queue.cc:237] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:15.206081 30345 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:05:15.206120 30475 raft_consensus.cc:1261] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:05:15.206385 15282 tablet_copy_client.cc:647] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:15.206444 15299 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:15.206646 14626 consensus_queue.cc:1035] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:15.207727 15282 tablet_copy_client.cc:523] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:15.208992 30345 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 6efe17dcca7b4babad4785d19552b699 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.208990 30475 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.209431 15282 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap starting.
I20240502 08:05:15.208640 15302 raft_consensus.cc:2932] T 21d939e5035541a4b003c5d276493d5c P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.211809 28249 catalog_manager.cc:5484] T 21d939e5035541a4b003c5d276493d5c P 29126d5aee924f81bdd47cc7153420a1 reported cstate change: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.215416 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 21d939e5035541a4b003c5d276493d5c with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39806
I20240502 08:05:15.215651 15177 tablet_replica.cc:330] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:05:15.215803 15177 raft_consensus.cc:2229] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:15.215965 15177 raft_consensus.cc:2258] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:15.216543 15177 ts_tablet_manager.cc:1875] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:15.219569 15177 ts_tablet_manager.cc:1888] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240502 08:05:15.219651 15177 log.cc:1196] T 21d939e5035541a4b003c5d276493d5c P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/21d939e5035541a4b003c5d276493d5c
I20240502 08:05:15.220122 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 21d939e5035541a4b003c5d276493d5c (table rebalance_test_table_3 [id=c7d0ad9cdc0b41d4926585d13b3e05ea]) successfully deleted
I20240502 08:05:15.224073 15282 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:15.224371 15282 tablet_bootstrap.cc:492] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: Bootstrap complete.
I20240502 08:05:15.224601 15282 ts_tablet_manager.cc:1367] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20240502 08:05:15.225109 15282 raft_consensus.cc:348] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:15.225216 15282 raft_consensus.cc:729] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29126d5aee924f81bdd47cc7153420a1, State: Initialized, Role: FOLLOWER
I20240502 08:05:15.225356 15282 consensus_queue.cc:260] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:15.225677 15282 ts_tablet_manager.cc:1398] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:15.226336 28419 tablet_copy_service.cc:342] P 71922d97e1a4414698ecd14a7b2dedec: Request end of tablet copy session 29126d5aee924f81bdd47cc7153420a1-3598dbda342f49aea2384b5beb17d7a1 received from {username='slave'} at 127.16.36.199:52255
I20240502 08:05:15.226411 28419 tablet_copy_service.cc:433] P 71922d97e1a4414698ecd14a7b2dedec: ending tablet copy session 29126d5aee924f81bdd47cc7153420a1-3598dbda342f49aea2384b5beb17d7a1 on tablet 3598dbda342f49aea2384b5beb17d7a1 with peer 29126d5aee924f81bdd47cc7153420a1
I20240502 08:05:15.282869 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.288162 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:15.294973 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:15.297029 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.305949 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.306052 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.307713 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.347010 30524 maintenance_manager.cc:419] P 29126d5aee924f81bdd47cc7153420a1: Scheduling UndoDeltaBlockGCOp(3598dbda342f49aea2384b5beb17d7a1): 4840 bytes on disk
I20240502 08:05:15.347504 30414 maintenance_manager.cc:643] P 29126d5aee924f81bdd47cc7153420a1: UndoDeltaBlockGCOp(3598dbda342f49aea2384b5beb17d7a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240502 08:05:15.366755 28529 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:15.367913 30475 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:15.368346 14692 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:15.370021 14697 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:15.371696 30475 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:15.372601 28251 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 7 to 8, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) evicted. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:15.376351 15362 ts_tablet_manager.cc:902] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:05:15.376560 28529 consensus_queue.cc:237] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:05:15.376940 15362 tablet_copy_client.cc:308] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:05:15.377828 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 3288b438832e488e8257227cb8934565 from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:47939)
I20240502 08:05:15.377928 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 3288b438832e488e8257227cb8934565 from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:47939: session id = 71922d97e1a4414698ecd14a7b2dedec-3288b438832e488e8257227cb8934565
W20240502 08:05:15.378226 28465 consensus_peers.cc:485] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e -> Peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): Couldn't send request to peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:15.378859 28548 tablet_copy_source_session.cc:205] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:05:15.379060 28771 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e1cd0ec72d1429f9621d217b1bf845a with delete_type TABLET_DATA_TOMBSTONED (TS de423605477a4819a9569821f59f03e4 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:34548
I20240502 08:05:15.379340 30475 raft_consensus.cc:1261] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:15.379439 15367 tablet_replica.cc:330] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: stopping tablet replica
I20240502 08:05:15.379580 15367 raft_consensus.cc:2229] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:15.379772 15367 raft_consensus.cc:2258] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:15.379871 14692 consensus_queue.cc:1035] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:15.380569 15362 ts_tablet_manager.cc:1875] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:15.380579 15367 ts_tablet_manager.cc:1875] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:15.381520 14697 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:05:15.382011 30475 raft_consensus.cc:2932] T 9e1cd0ec72d1429f9621d217b1bf845a P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:05:15.382758 15362 ts_tablet_manager.cc:1888] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240502 08:05:15.382925 15362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3288b438832e488e8257227cb8934565. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:15.384347 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 9e1cd0ec72d1429f9621d217b1bf845a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:15.384719 28251 catalog_manager.cc:5484] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 8 to 9, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) added. New cstate: current_term: 1 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } }
I20240502 08:05:15.387809 15362 tablet_copy_client.cc:783] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 5 data blocks...
I20240502 08:05:15.389112 15367 ts_tablet_manager.cc:1888] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240502 08:05:15.389192 15367 log.cc:1196] T 9e1cd0ec72d1429f9621d217b1bf845a P de423605477a4819a9569821f59f03e4: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e1cd0ec72d1429f9621d217b1bf845a
I20240502 08:05:15.389727 28244 catalog_manager.cc:4830] TS de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): tablet 9e1cd0ec72d1429f9621d217b1bf845a (table rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]) successfully deleted
I20240502 08:05:15.393114 15362 tablet_copy_client.cc:647] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:15.394529 15362 tablet_copy_client.cc:523] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:15.396165 15362 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:05:15.406313 15362 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:15.406606 15362 tablet_bootstrap.cc:492] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:05:15.406795 15362 ts_tablet_manager.cc:1367] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 08:05:15.407277 15362 raft_consensus.cc:348] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:15.407394 15362 raft_consensus.cc:729] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:05:15.407526 15362 consensus_queue.cc:260] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:15.407802 15362 ts_tablet_manager.cc:1398] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:05:15.408329 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-3288b438832e488e8257227cb8934565 received from {username='slave'} at 127.16.36.193:47939
I20240502 08:05:15.408412 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-3288b438832e488e8257227cb8934565 on tablet 3288b438832e488e8257227cb8934565 with peer 71922d97e1a4414698ecd14a7b2dedec
I20240502 08:05:15.410310 28529 raft_consensus.cc:1203] T f98044a1730e4a04a7088199d4f0850e P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:05:15.459249 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:15.464145 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.465140 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.467972 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.472116 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.477795 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:15.500016 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.571985 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565): 4840 bytes on disk
I20240502 08:05:15.574293 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(3288b438832e488e8257227cb8934565) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
I20240502 08:05:15.591017 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:15.593328 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.593720 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.597415 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:15.597822 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.607882 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.614876 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.649077 30475 raft_consensus.cc:1203] T 3598dbda342f49aea2384b5beb17d7a1 P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:05:15.658272 28661 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e"
dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50770
I20240502 08:05:15.658421 28661 raft_consensus.cc:595] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:15.661041 14697 raft_consensus.cc:998] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.031s)
I20240502 08:05:15.661408 14697 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 4.10, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:05:15.662552 30213 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 4 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 4 index: 10. Preceding OpId from leader: term: 4 index: 11. (index mismatch)
I20240502 08:05:15.662925 14692 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:15.664647 14697 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 4 LEADER]: Committing config change with OpId 4.11: config changed from index 9 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:05:15.665123 30213 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 4 FOLLOWER]: Committing config change with OpId 4.11: config changed from index 9 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:05:15.667249 28248 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 9 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 4 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:05:15.670667 28529 consensus_queue.cc:237] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 4.11, Last appended by leader: 9, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:15.671725 30213 raft_consensus.cc:1261] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 4 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 4 index: 11. Preceding OpId from leader: term: 4 index: 12. (index mismatch)
W20240502 08:05:15.672037 28464 consensus_peers.cc:485] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): Couldn't send request to peer 71922d97e1a4414698ecd14a7b2dedec. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:15.672081 14697 consensus_queue.cc:1035] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:05:15.673550 14692 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e [term 4 LEADER]: Committing config change with OpId 4.12: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:15.673996 30213 raft_consensus.cc:2932] T c65fe7f823df4b4996769fc784d3adb7 P 6c68c916f39a45f697df7eaf8973225a [term 4 FOLLOWER]: Committing config change with OpId 4.12: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:15.675145 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet c65fe7f823df4b4996769fc784d3adb7 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:15.676034 28248 catalog_manager.cc:5484] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 11 to 12, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) added. New cstate: current_term: 4 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
W20240502 08:05:15.719832 28241 catalog_manager.cc:4573] TS e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 0499a3f3e6bd4069bf992e32f1abfdb0: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
I20240502 08:05:15.725531 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.729570 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:15.730417 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.735836 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.745313 15309 raft_consensus.cc:998] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.346s)
I20240502 08:05:15.745697 15309 consensus_queue.cc:237] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:15.747179 30213 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:05:15.747581 15001 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:15.748715 28398 raft_consensus.cc:1261] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 08:05:15.749080 15001 consensus_queue.cc:1035] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:15.749763 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.751260 15001 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:15.751574 28398 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:15.753160 30215 raft_consensus.cc:2932] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:15.755954 28252 catalog_manager.cc:5484] T 698f494a22d34e659766b97cb24d0293 P 6c68c916f39a45f697df7eaf8973225a reported cstate change: config changed from index 10 to 11, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:15.757292 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
W20240502 08:05:15.773489 28594 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111) [suppressed 35 similar messages]
I20240502 08:05:15.774873 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.803277 28398 raft_consensus.cc:1203] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20240502 08:05:15.872870 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:15.873138 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.883245 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:15.893451 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.898561 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:15.902109 28378 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:15.916777 15551 ts_tablet_manager.cc:902] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:05:15.917379 15504 raft_consensus.cc:979] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: : Instructing follower 71922d97e1a4414698ecd14a7b2dedec to start an election
I20240502 08:05:15.917496 15504 raft_consensus.cc:1067] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Signalling peer 71922d97e1a4414698ecd14a7b2dedec to start an election
I20240502 08:05:15.917515 15551 tablet_copy_client.cc:308] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:05:15.917990 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet 9e1cd0ec72d1429f9621d217b1bf845a from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 ({username='slave'} at 127.16.36.195:46469)
I20240502 08:05:15.918108 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet 9e1cd0ec72d1429f9621d217b1bf845a from peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44 at {username='slave'} at 127.16.36.195:46469: session id = ed3ca7e829bc42b5a8e9a7d1bdcc5a44-9e1cd0ec72d1429f9621d217b1bf845a
I20240502 08:05:15.919085 28548 tablet_copy_source_session.cc:205] T 9e1cd0ec72d1429f9621d217b1bf845a P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:05:15.919656 28398 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e"
dest_uuid: "71922d97e1a4414698ecd14a7b2dedec"
from {username='slave'} at 127.16.36.195:55199
I20240502 08:05:15.919812 28398 raft_consensus.cc:482] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:15.919879 28398 raft_consensus.cc:3037] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:15.920753 28641 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:15.920958 28398 raft_consensus.cc:504] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:15.921332 28398 leader_election.cc:288] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 2 election: Requested vote from peers ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991), 29126d5aee924f81bdd47cc7153420a1 (127.16.36.199:43589)
I20240502 08:05:15.921984 30475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "29126d5aee924f81bdd47cc7153420a1"
I20240502 08:05:15.922109 30475 raft_consensus.cc:3037] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:15.923168 30475 raft_consensus.cc:2454] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 2.
I20240502 08:05:15.924531 28332 leader_election.cc:302] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29126d5aee924f81bdd47cc7153420a1, 71922d97e1a4414698ecd14a7b2dedec; no voters:
I20240502 08:05:15.924726 15071 raft_consensus.cc:2781] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:15.924808 15071 raft_consensus.cc:686] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Becoming Leader. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Running, Role: LEADER
I20240502 08:05:15.924947 15071 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:15.927899 28252 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: term changed from 1 to 2, leader changed from ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) to 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193). New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:15.928625 28661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cb29b8b566747e5b82a8969a070123e" candidate_uuid: "71922d97e1a4414698ecd14a7b2dedec" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44"
I20240502 08:05:15.928752 28661 raft_consensus.cc:3032] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:15.928790 28661 raft_consensus.cc:729] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Running, Role: LEADER
I20240502 08:05:15.928922 28661 consensus_queue.cc:260] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:15.929082 28661 raft_consensus.cc:3037] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:15.930141 28661 raft_consensus.cc:2454] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71922d97e1a4414698ecd14a7b2dedec in term 2.
I20240502 08:05:15.930660 15551 ts_tablet_manager.cc:1875] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:15.933080 15551 ts_tablet_manager.cc:1888] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:15.933260 15551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e1cd0ec72d1429f9621d217b1bf845a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:15.934401 15551 tablet_copy_client.cc:783] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 5 data blocks...
I20240502 08:05:15.940958 15551 tablet_copy_client.cc:647] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:15.942379 15551 tablet_copy_client.cc:523] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:15.944378 15551 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap starting.
I20240502 08:05:15.955265 15551 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:15.955580 15551 tablet_bootstrap.cc:492] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Bootstrap complete.
I20240502 08:05:15.955785 15551 ts_tablet_manager.cc:1367] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 08:05:15.956189 15551 raft_consensus.cc:348] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:05:15.956295 15551 raft_consensus.cc:729] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed3ca7e829bc42b5a8e9a7d1bdcc5a44, State: Initialized, Role: FOLLOWER
I20240502 08:05:15.956497 15551 consensus_queue.cc:260] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } }
I20240502 08:05:15.956784 15551 ts_tablet_manager.cc:1398] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:05:15.957347 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-9e1cd0ec72d1429f9621d217b1bf845a received from {username='slave'} at 127.16.36.195:46469
I20240502 08:05:15.957424 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session ed3ca7e829bc42b5a8e9a7d1bdcc5a44-9e1cd0ec72d1429f9621d217b1bf845a on tablet 9e1cd0ec72d1429f9621d217b1bf845a with peer ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:05:15.971179 28529 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }
I20240502 08:05:15.972504 30345 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:05:15.973075 14697 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:15.975041 14692 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.975478 30345 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.978262 28252 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 10 to 11, VOTER 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } }
I20240502 08:05:15.982939 28378 tablet_service.cc:1514] Processing DeleteTablet for tablet 3288b438832e488e8257227cb8934565 with delete_type TABLET_DATA_TOMBSTONED (TS 71922d97e1a4414698ecd14a7b2dedec not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39806
I20240502 08:05:15.983299 15580 tablet_replica.cc:330] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: stopping tablet replica
I20240502 08:05:15.983510 15580 raft_consensus.cc:2229] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:15.983664 15580 raft_consensus.cc:2258] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:15.984022 28528 consensus_queue.cc:237] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:05:15.984305 15580 ts_tablet_manager.cc:1875] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:15.985594 30345 raft_consensus.cc:1261] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Refusing update from remote peer c39bcb719db947b8afb0bb141e1f7a4e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240502 08:05:15.985870 28466 consensus_peers.cc:485] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3288b438832e488e8257227cb8934565. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:15.986169 14697 consensus_queue.cc:1035] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 08:05:15.987877 15580 ts_tablet_manager.cc:1888] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:05:15.987957 15580 log.cc:1196] T 3288b438832e488e8257227cb8934565 P 71922d97e1a4414698ecd14a7b2dedec: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/3288b438832e488e8257227cb8934565
I20240502 08:05:15.988652 28242 catalog_manager.cc:4830] TS 71922d97e1a4414698ecd14a7b2dedec (127.16.36.193:33537): tablet 3288b438832e488e8257227cb8934565 (table rebalance_test_table_2 [id=b3f6d9e649424514a7ccda14b8f06c8c]) successfully deleted
I20240502 08:05:15.989179 14692 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:05:15.989727 30345 raft_consensus.cc:2932] T 3288b438832e488e8257227cb8934565 P 6efe17dcca7b4babad4785d19552b699 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:05:15.990868 28241 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 3288b438832e488e8257227cb8934565 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:15.992121 28252 catalog_manager.cc:5484] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e reported cstate change: config changed from index 11 to 12, VOTER 6c68c916f39a45f697df7eaf8973225a (127.16.36.197) added. New cstate: current_term: 2 leader_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } }
I20240502 08:05:16.002535 28708 maintenance_manager.cc:419] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Scheduling UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a): 4840 bytes on disk
I20240502 08:05:16.002998 28598 maintenance_manager.cc:643] P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: UndoDeltaBlockGCOp(9e1cd0ec72d1429f9621d217b1bf845a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240502 08:05:16.053392 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.055977 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.056963 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.060164 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.063465 28641 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:16.071239 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:16.083024 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:16.185317 15677 ts_tablet_manager.cc:902] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
I20240502 08:05:16.185946 15677 tablet_copy_client.cc:308] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:40581
I20240502 08:05:16.186450 28548 tablet_copy_service.cc:140] P c39bcb719db947b8afb0bb141e1f7a4e: Received BeginTabletCopySession request for tablet c65fe7f823df4b4996769fc784d3adb7 from peer 71922d97e1a4414698ecd14a7b2dedec ({username='slave'} at 127.16.36.193:47939)
I20240502 08:05:16.186558 28548 tablet_copy_service.cc:161] P c39bcb719db947b8afb0bb141e1f7a4e: Beginning new tablet copy session on tablet c65fe7f823df4b4996769fc784d3adb7 from peer 71922d97e1a4414698ecd14a7b2dedec at {username='slave'} at 127.16.36.193:47939: session id = 71922d97e1a4414698ecd14a7b2dedec-c65fe7f823df4b4996769fc784d3adb7
I20240502 08:05:16.187762 28548 tablet_copy_source_session.cc:205] T c65fe7f823df4b4996769fc784d3adb7 P c39bcb719db947b8afb0bb141e1f7a4e: Tablet Copy: opened 5 blocks and 1 log segments
I20240502 08:05:16.189532 15677 ts_tablet_manager.cc:1875] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:16.192081 15677 ts_tablet_manager.cc:1888] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 3.6
I20240502 08:05:16.192292 15677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c65fe7f823df4b4996769fc784d3adb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:16.193948 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.194334 15677 tablet_copy_client.cc:783] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 5 data blocks...
I20240502 08:05:16.198468 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.202191 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:16.202276 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:16.205183 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.206552 28641 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:16.207418 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.209053 15677 tablet_copy_client.cc:647] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:16.215068 15677 tablet_copy_client.cc:523] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:16.216692 15677 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap starting.
I20240502 08:05:16.255852 15677 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:16.256237 15677 tablet_bootstrap.cc:492] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Bootstrap complete.
I20240502 08:05:16.256392 15677 ts_tablet_manager.cc:1367] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Time spent bootstrapping tablet: real 0.040s user 0.011s sys 0.000s
I20240502 08:05:16.256785 15677 raft_consensus.cc:348] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 4 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:16.256892 15677 raft_consensus.cc:729] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71922d97e1a4414698ecd14a7b2dedec, State: Initialized, Role: FOLLOWER
I20240502 08:05:16.257014 15677 consensus_queue.cc:260] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 4.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:16.257282 15677 ts_tablet_manager.cc:1398] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:16.257835 28548 tablet_copy_service.cc:342] P c39bcb719db947b8afb0bb141e1f7a4e: Request end of tablet copy session 71922d97e1a4414698ecd14a7b2dedec-c65fe7f823df4b4996769fc784d3adb7 received from {username='slave'} at 127.16.36.193:47939
I20240502 08:05:16.257923 28548 tablet_copy_service.cc:433] P c39bcb719db947b8afb0bb141e1f7a4e: ending tablet copy session 71922d97e1a4414698ecd14a7b2dedec-c65fe7f823df4b4996769fc784d3adb7 on tablet c65fe7f823df4b4996769fc784d3adb7 with peer 71922d97e1a4414698ecd14a7b2dedec
I20240502 08:05:16.327626 28446 maintenance_manager.cc:419] P 71922d97e1a4414698ecd14a7b2dedec: Scheduling UndoDeltaBlockGCOp(c65fe7f823df4b4996769fc784d3adb7): 4586 bytes on disk
I20240502 08:05:16.328218 28336 maintenance_manager.cc:643] P 71922d97e1a4414698ecd14a7b2dedec: UndoDeltaBlockGCOp(c65fe7f823df4b4996769fc784d3adb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240502 08:05:16.330821 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.334074 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.343784 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:16.343941 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:16.344381 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.345808 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.357576 28641 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:16.396107 15299 raft_consensus.cc:998] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4: Attempting to remove follower e5e9553dabc94f218f6427c1685ffc38 from the Raft config. Reason: Leader has been unable to successfully communicate with peer e5e9553dabc94f218f6427c1685ffc38 for more than 300 seconds (300.125s)
I20240502 08:05:16.396598 15299 consensus_queue.cc:237] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } }
I20240502 08:05:16.398272 28528 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:16.398453 28398 raft_consensus.cc:1261] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Refusing update from remote peer de423605477a4819a9569821f59f03e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:16.398927 15302 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:16.399175 15738 consensus_queue.cc:1035] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:16.400787 15302 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:16.401197 28398 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P 71922d97e1a4414698ecd14a7b2dedec [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:16.403322 28250 catalog_manager.cc:5484] T fddc382ab3164ccdaa86f802c83da8be P de423605477a4819a9569821f59f03e4 reported cstate change: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New cstate: current_term: 1 leader_uuid: "de423605477a4819a9569821f59f03e4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:16.403455 28528 raft_consensus.cc:2932] T fddc382ab3164ccdaa86f802c83da8be P c39bcb719db947b8afb0bb141e1f7a4e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } }
I20240502 08:05:16.413718 28661 raft_consensus.cc:1203] T 9e1cd0ec72d1429f9621d217b1bf845a P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240502 08:05:16.441728 30475 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:16.442687 15458 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:16.445484 28661 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:16.445869 15769 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:16.469545 30194 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.474761 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.475417 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:16.486838 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.489632 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.492520 28641 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:16.501470 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:16.552598 28397 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } }
I20240502 08:05:16.553675 30475 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:16.554049 15769 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:16.555464 15458 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:16.555899 30475 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:16.558219 28252 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 7 to 9, VOTER ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } }
I20240502 08:05:16.561079 28641 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cb29b8b566747e5b82a8969a070123e with delete_type TABLET_DATA_TOMBSTONED (TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:50666
I20240502 08:05:16.561503 15801 tablet_replica.cc:330] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: stopping tablet replica
I20240502 08:05:16.561662 15801 raft_consensus.cc:2229] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:16.561774 15801 raft_consensus.cc:2258] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:16.561837 28397 consensus_queue.cc:237] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } }
I20240502 08:05:16.562641 15801 ts_tablet_manager.cc:1875] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:16.563338 30475 raft_consensus.cc:1261] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Refusing update from remote peer 71922d97e1a4414698ecd14a7b2dedec: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240502 08:05:16.563357 28335 consensus_peers.cc:485] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec -> Peer de423605477a4819a9569821f59f03e4 (127.16.36.196:33237): Couldn't send request to peer de423605477a4819a9569821f59f03e4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4cb29b8b566747e5b82a8969a070123e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:16.564008 15458 consensus_queue.cc:1035] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [LEADER]: Connected to new peer: Peer: permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:16.565904 15769 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:05:16.566365 15801 ts_tablet_manager.cc:1888] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240502 08:05:16.566452 15801 log.cc:1196] T 4cb29b8b566747e5b82a8969a070123e P ed3ca7e829bc42b5a8e9a7d1bdcc5a44: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/4cb29b8b566747e5b82a8969a070123e
I20240502 08:05:16.566455 30475 raft_consensus.cc:2932] T 4cb29b8b566747e5b82a8969a070123e P 29126d5aee924f81bdd47cc7153420a1 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:05:16.567157 28243 catalog_manager.cc:4830] TS ed3ca7e829bc42b5a8e9a7d1bdcc5a44 (127.16.36.195:33991): tablet 4cb29b8b566747e5b82a8969a070123e (table rebalance_test_table_7 [id=f8cc28afdd9143869e64c179746de6ab]) successfully deleted
I20240502 08:05:16.567466 28242 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:VOTER RPC for tablet 4cb29b8b566747e5b82a8969a070123e with cas_config_opid_index 9: ChangeConfig:ADD_PEER:VOTER succeeded (attempt 1)
I20240502 08:05:16.569015 28253 catalog_manager.cc:5484] T 4cb29b8b566747e5b82a8969a070123e P 71922d97e1a4414698ecd14a7b2dedec reported cstate change: config changed from index 9 to 10, VOTER de423605477a4819a9569821f59f03e4 (127.16.36.196) added. New cstate: current_term: 2 leader_uuid: "71922d97e1a4414698ecd14a7b2dedec" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "29126d5aee924f81bdd47cc7153420a1" member_type: VOTER last_known_addr { host: "127.16.36.199" port: 43589 } } peers { permanent_uuid: "de423605477a4819a9569821f59f03e4" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 33237 } } }
I20240502 08:05:16.580585 15807 ts_tablet_manager.cc:902] T 3288b438832e488e8257227cb8934565 P 6c68c916f39a45f697df7eaf8973225a: Initiating tablet copy from peer c39bcb719db947b8afb0bb141e1f7a4e (127.16.36.194:40581)
F20240502 08:05:16.580811 15807 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.queue_time_us, tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.queue_time_us) (tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_cpu_time_us, tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_cpu_time_us) (tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_wall_time_us, tablet-download-pool-0499a3f3e6bd4069bf992e32f1abfdb0.run_wall_time_us) (tablet-download-pool-04d358245cdf4b2ebb0b5005f75cf1e0.queue_time_us, tablet-download-pool-04d358245cdf4b2ebb0b5005f75cf1e0.queue_time_us) (tablet-download-pool-04d358245cdf4b2ebb0b5005f75cf1e0.run_cpu_time_us, tablet-download-pool-04d358245cdf4b2ebb0b5005f75cf1e0.run_cpu_time_us) (tablet-download-pool-04d358245cdf4b2ebb0b5005f75cf1e0.run_wall_time_us, tablet-download-pool-04d358245cdf4b2ebb0b5005f75cf1e0.run_wall_time_us) (tablet-download-pool-14bbfb3758e04aeab0df4bbab8380061.queue_time_us, tablet-download-pool-14bbfb3758e04aeab0df4bbab8380061.queue_time_us) (tablet-download-pool-14bbfb3758e04aeab0df4bbab8380061.run_cpu_time_us, tablet-download-pool-14bbfb3758e04aeab0df4bbab8380061.run_cpu_time_us) (tablet-download-pool-14bbfb3758e04aeab0df4bbab8380061.run_wall_time_us, tablet-download-pool-14bbfb3758e04aeab0df4bbab8380061.run_wall_time_us) (tablet-download-pool-2ba62cb7ab7b429d85e0ac037c127cb4.queue_time_us, tablet-download-pool-2ba62cb7ab7b429d85e0ac037c127cb4.queue_time_us) (tablet-download-pool-2ba62cb7ab7b429d85e0ac037c127cb4.run_cpu_time_us, tablet-download-pool-2ba62cb7ab7b429d85e0ac037c127cb4.run_cpu_time_us) (tablet-download-pool-2ba62cb7ab7b429d85e0ac037c127cb4.run_wall_time_us, tablet-download-pool-2ba62cb7ab7b429d85e0ac037c127cb4.run_wall_time_us) (tablet-download-pool-3288b438832e488e8257227cb8934565.queue_time_us, tablet-download-pool-3288b438832e488e8257227cb8934565.queue_time_us) (tablet-download-pool-5d1c1fa046b741aa8cf296aa57847fe3.queue_time_us, tablet-download-pool-5d1c1fa046b741aa8cf296aa57847fe3.queue_time_us) (tablet-download-pool-5d1c1fa046b741aa8cf296aa57847fe3.run_cpu_time_us, tablet-download-pool-5d1c1fa046b741aa8cf296aa57847fe3.run_cpu_time_us) (tablet-download-pool-5d1c1fa046b741aa8cf296aa57847fe3.run_wall_time_us, tablet-download-pool-5d1c1fa046b741aa8cf296aa57847fe3.run_wall_time_us) (tablet-download-pool-698f494a22d34e659766b97cb24d0293.queue_time_us, tablet-download-pool-698f494a22d34e659766b97cb24d0293.queue_time_us) (tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_cpu_time_us, tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_cpu_time_us) (tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_wall_time_us, tablet-download-pool-698f494a22d34e659766b97cb24d0293.run_wall_time_us) (tablet-download-pool-9c7e7dadff0a4cd08deb043a37a54424.queue_time_us, tablet-download-pool-9c7e7dadff0a4cd08deb043a37a54424.queue_time_us) (tablet-download-pool-9c7e7dadff0a4cd08deb043a37a54424.run_cpu_time_us, tablet-download-pool-9c7e7dadff0a4cd08deb043a37a54424.run_cpu_time_us) (tablet-download-pool-9c7e7dadff0a4cd08deb043a37a54424.run_wall_time_us, tablet-download-pool-9c7e7dadff0a4cd08deb043a37a54424.run_wall_time_us) (tablet-download-pool-a28d406b43194bc6b775707d021618f0.queue_time_us, tablet-download-pool-a28d406b43194bc6b775707d021618f0.queue_time_us) (tablet-download-pool-a28d406b43194bc6b775707d021618f0.run_cpu_time_us, tablet-download-pool-a28d406b43194bc6b775707d021618f0.run_cpu_time_us) (tablet-download-pool-a28d406b43194bc6b775707d021618f0.run_wall_time_us, tablet-download-pool-a28d406b43194bc6b775707d021618f0.run_wall_time_us) (tablet-download-pool-afc9c4d9dd5b4c039686ccd918616859.queue_time_us, tablet-download-pool-afc9c4d9dd5b4c039686ccd918616859.queue_time_us) (tablet-download-pool-afc9c4d9dd5b4c039686ccd918616859.run_cpu_time_us, tablet-download-pool-afc9c4d9dd5b4c039686ccd918616859.run_cpu_time_us) (tablet-download-pool-afc9c4d9dd5b4c039686ccd918616859.run_wall_time_us, tablet-download-pool-afc9c4d9dd5b4c039686ccd918616859.run_wall_time_us) (tablet-download-pool-c47d608895f94a2c82e695ca16677abf.queue_time_us, tablet-download-pool-c47d608895f94a2c82e695ca16677abf.queue_time_us) (tablet-download-pool-c47d608895f94a2c82e695ca16677abf.run_cpu_time_us, tablet-download-pool-c47d608895f94a2c82e695ca16677abf.run_cpu_time_us) (tablet-download-pool-c47d608895f94a2c82e695ca16677abf.run_wall_time_us, tablet-download-pool-c47d608895f94a2c82e695ca16677abf.run_wall_time_us) (tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.queue_time_us, tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.queue_time_us) (tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_cpu_time_us, tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_cpu_time_us) (tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_wall_time_us, tablet-download-pool-c65fe7f823df4b4996769fc784d3adb7.run_wall_time_us) (tablet-download-pool-dcd68a21ed7b4205a2f0cf75f80df0e6.queue_time_us, tablet-download-pool-dcd68a21ed7b4205a2f0cf75f80df0e6.queue_time_us) (tablet-download-pool-dcd68a21ed7b4205a2f0cf75f80df0e6.run_cpu_time_us, tablet-download-pool-dcd68a21ed7b4205a2f0cf75f80df0e6.run_cpu_time_us) (tablet-download-pool-dcd68a21ed7b4205a2f0cf75f80df0e6.run_wall_time_us, tablet-download-pool-dcd68a21ed7b4205a2f0cf75f80df0e6.run_wall_time_us) (tablet-download-pool-de90ee4ef90e4cb69ec37d7d9f0c86ef.queue_time_us, tablet-download-pool-de90ee4ef90e4cb69ec37d7d9f0c86ef.queue_time_us) (tablet-download-pool-de90ee4ef90e4cb69ec37d7d9f0c86ef.run_cpu_time_us, tablet-download-pool-de90ee4ef90e4cb69ec37d7d9f0c86ef.run_cpu_time_us) (tablet-download-pool-de90ee4ef90e4cb69ec37d7d9f0c86ef.run_wall_time_us, tablet-download-pool-de90ee4ef90e4cb69ec37d7d9f0c86ef.run_wall_time_us) (tablet-download-pool-df20642aef7748ff8962b3095e1ab56c.queue_time_us, tablet-download-pool-df20642aef7748ff8962b3095e1ab56c.queue_time_us) (tablet-download-pool-df20642aef7748ff8962b3095e1ab56c.run_cpu_time_us, tablet-download-pool-df20642aef7748ff8962b3095e1ab56c.run_cpu_time_us) (tablet-download-pool-df20642aef7748ff8962b3095e1ab56c.run_wall_time_us, tablet-download-pool-df20642aef7748ff8962b3095e1ab56c.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fe068418dcd google::LogMessage::Fail() at ??:0
@ 0x7fe06841cb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fe0684187cc google::LogMessage::Flush() at ??:0
@ 0x7fe068419f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe069882044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fe069866f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fe069864efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fe06c41777a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fe06c420a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fe06c4a176f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fe06c4b54f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fe06c4a0226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fe06c4afc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559921f8c75c std::function<>::operator()() at ??:0
@ 0x7fe069869e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe06986a6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe06986c03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x559921f8c75c std::function<>::operator()() at ??:0
@ 0x7fe069859807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe06a0976db start_thread at ??:0
@ 0x7fe0675cf71f clone at ??:0
I20240502 08:05:16.740528 28397 raft_consensus.cc:1203] T c65fe7f823df4b4996769fc784d3adb7 P 71922d97e1a4414698ecd14a7b2dedec [term 4 FOLLOWER]: Deduplicated request from leader. Original: 4.11->[4.12-4.12] Dedup: 4.12->[]
W20240502 08:05:16.765873 28241 catalog_manager.cc:4573] TS e5e9553dabc94f218f6427c1685ffc38 (127.16.36.200:43189): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3598dbda342f49aea2384b5beb17d7a1: Network error: Client connection negotiation failed: client connection to 127.16.36.200:43189: connect: Connection refused (error 111)
W20240502 08:05:16.815176 30413 connection.cc:535] client connection to 127.16.36.197:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.815227 30282 connection.cc:535] client connection to 127.16.36.197:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.815351 30413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.815367 30282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.815709 28728 connection.cc:535] client connection to 127.16.36.197:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.815829 28335 connection.cc:535] client connection to 127.16.36.197:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.815939 28597 connection.cc:535] client connection to 127.16.36.197:35407 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:16.816689 28728 consensus_peers.cc:485] T 14bbfb3758e04aeab0df4bbab8380061 P de423605477a4819a9569821f59f03e4 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:16.817153 30413 consensus_peers.cc:485] T a28d406b43194bc6b775707d021618f0 P 29126d5aee924f81bdd47cc7153420a1 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:16.817236 30413 consensus_peers.cc:485] T 04d358245cdf4b2ebb0b5005f75cf1e0 P 29126d5aee924f81bdd47cc7153420a1 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:16.817679 28335 consensus_peers.cc:485] T c47d608895f94a2c82e695ca16677abf P 71922d97e1a4414698ecd14a7b2dedec -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:16.817955 28597 consensus_peers.cc:485] T df20642aef7748ff8962b3095e1ab56c P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:16.818050 28597 consensus_peers.cc:485] T 698f494a22d34e659766b97cb24d0293 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 08:05:16.818143 28466 consensus_peers.cc:462] T 3288b438832e488e8257227cb8934565 P c39bcb719db947b8afb0bb141e1f7a4e -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111)
W20240502 08:05:16.818174 30282 consensus_peers.cc:485] T 9c7e7dadff0a4cd08deb043a37a54424 P 6efe17dcca7b4babad4785d19552b699 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:16.863878 30456 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.864415 28378 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:16.865756 30325 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:16.873049 28771 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:16.877192 28510 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:16.879791 28641 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 08:05:16.921667 30282 consensus_peers.cc:485] T afc9c4d9dd5b4c039686ccd918616859 P 6efe17dcca7b4babad4785d19552b699 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:16.929275 30660 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240502 07:59:58.396427 30663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 07:59:58.396806 30663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 07:59:58.401404 30663 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 07:59:59.532778 30663 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 07:59:59.954993 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.955205 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.955276 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.955329 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.955368 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.955401 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.955437 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.955475 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 07:59:59.955525 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.955569 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.955621 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 07:59:59.955670 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 07:59:59.955729 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 07:59:59.955777 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
I20240502 08:00:00.128206 30663 rebalancer_tool.cc:1074] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:00.248034 30663 rebalancer_tool.cc:1074] tablet c47d608895f94a2c82e695ca16677abf: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.368539 30663 rebalancer_tool.cc:1074] tablet 5d1c1fa046b741aa8cf296aa57847fe3: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:00.440739 30663 rebalancer_tool.cc:1074] tablet c65fe7f823df4b4996769fc784d3adb7: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:00.550727 30663 rebalancer_tool.cc:1074] tablet a28d406b43194bc6b775707d021618f0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:00.646824 30663 rebalancer_tool.cc:1074] tablet 698f494a22d34e659766b97cb24d0293: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move scheduled
I20240502 08:00:00.716877 30663 rebalancer_tool.cc:1084] tablet de90ee4ef90e4cb69ec37d7d9f0c86ef: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c39bcb719db947b8afb0bb141e1f7a4e" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 40581 } } peers { permanent_uuid: "71922d97e1a4414698ecd14a7b2dedec" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 33537 } } peers { permanent_uuid: "ed3ca7e829bc42b5a8e9a7d1bdcc5a44" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 33991 } } peers { permanent_uuid: "6efe17dcca7b4babad4785d19552b699" member_type: VOTER last_known_addr { host: "127.16.36.198" port: 45421 } } peers { permanent_uuid: "6c68c916f39a45f697df7eaf8973225a" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 35407 } }
I20240502 08:00:00.716965 30663 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:01.312001 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.312075 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.312103 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.312127 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.312177 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.312219 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.312265 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.312312 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.312338 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:01.312366 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.312390 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.312419 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.312491 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.312525 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.312553 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.312582 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:01.312610 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.312635 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:01.312660 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 172eddf6e26644619807f0650e92ccb4: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:01.312701 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:01.428305 30663 rebalancer_tool.cc:1074] tablet 14bbfb3758e04aeab0df4bbab8380061: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:01.636152 30663 rebalancer_tool.cc:1074] tablet fddc382ab3164ccdaa86f802c83da8be: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:01.645375 30663 rebalancer_tool.cc:1084] tablet 2ba62cb7ab7b429d85e0ac037c127cb4: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move ignored: Illegal state: tablet 2ba62cb7ab7b429d85e0ac037c127cb4: replica 6c68c916f39a45f697df7eaf8973225a is already present
I20240502 08:00:01.645478 30663 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:02.096859 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.097030 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.097102 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b3f6d9e649424514a7ccda14b8f06c8c: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.097163 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.097216 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.097293 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 1e2b1140681b44a284e5bcb21ee79110: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.108834 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.109009 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 77d8070fb35f4aeaa0aba32c29ec61cf: could not find any suitable replica to move from server 6c68c916f39a45f697df7eaf8973225a to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.109112 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e0a255a457004e77810fb9dc36cc3f28: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6c68c916f39a45f697df7eaf8973225a
W20240502 08:00:02.109181 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f22192bf738e4f63a739dd5907881674: could not find any suitable replica to move from server de423605477a4819a9569821f59f03e4 to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.109230 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server c39bcb719db947b8afb0bb141e1f7a4e to server 6efe17dcca7b4babad4785d19552b699
W20240502 08:00:02.109405 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 66e1eb2ffd3a40e499cba7cbdd607f68: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server e5e9553dabc94f218f6427c1685ffc38
W20240502 08:00:02.109501 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c7d0ad9cdc0b41d4926585d13b3e05ea: could not find any suitable replica to move from server 71922d97e1a4414698ecd14a7b2dedec to server 29126d5aee924f81bdd47cc7153420a1
W20240502 08:00:02.109571 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server ed3ca7e829bc42b5a8e9a7d1bdcc5a44 to server 6efe17dcca7b4babad4785d19552b699
I20240502 08:00:02.142119 30663 rebalancer_tool.cc:1074] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:02.218634 30663 rebalancer_tool.cc:1074] tablet 57f7c358ace742649df52380939ece9d: 'e5e9553dabc94f218f6427c1685ffc38' -> '' move scheduled
I20240502 08:00:02.371853 30663 rebalancer_tool.cc:1074] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:02.408730 30663 rebalancer_tool.cc:1074] tablet 7dbf7921ae84476ab53fbd4707f463eb: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:02.519248 30663 rebalancer_tool.cc:1074] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:02.607262 30663 rebalancer_tool.cc:1074] tablet 6a40382745f84f0eadf9e0ee2e057289: 'c39bcb719db947b8afb0bb141e1f7a4e' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:07.384714 30663 rebalancer_tool.cc:1136] tablet c47d608895f94a2c82e695ca16677abf: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:07.398213 30663 rebalancer_tool.cc:1136] tablet d08ad2a981fc4c13bc89d22feea7d556: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:09.555611 30663 rebalancer_tool.cc:1136] tablet 57f7c358ace742649df52380939ece9d: 'e5e9553dabc94f218f6427c1685ffc38' -> '' move completed: OK
I20240502 08:00:09.587203 30663 rebalancer_tool.cc:1136] tablet 698f494a22d34e659766b97cb24d0293: 'de423605477a4819a9569821f59f03e4' -> '6c68c916f39a45f697df7eaf8973225a' move completed: OK
I20240502 08:00:09.606226 30663 rebalancer_tool.cc:1074] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:10.269623 30663 rebalancer_tool.cc:1136] tablet 3288b438832e488e8257227cb8934565: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:10.288664 30663 rebalancer_tool.cc:1136] tablet 9c7e7dadff0a4cd08deb043a37a54424: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:10.303839 30663 rebalancer_tool.cc:1136] tablet 7dbf7921ae84476ab53fbd4707f463eb: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:00:10.983090 30663 rebalancer_tool.cc:1136] tablet 167b5f3b818048af81b1350839ebe489: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:10.999714 30663 rebalancer_tool.cc:1136] tablet 14bbfb3758e04aeab0df4bbab8380061: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:11.021481 30663 rebalancer_tool.cc:1136] tablet fddc382ab3164ccdaa86f802c83da8be: '71922d97e1a4414698ecd14a7b2dedec' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:11.810649 30663 rebalancer_tool.cc:1136] tablet 5d1c1fa046b741aa8cf296aa57847fe3: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:00:11.834146 30663 rebalancer_tool.cc:1136] tablet a28d406b43194bc6b775707d021618f0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:11.864982 30663 rebalancer_tool.cc:1074] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:11.880260 30663 rebalancer_tool.cc:1074] tablet 3598dbda342f49aea2384b5beb17d7a1: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move scheduled
I20240502 08:00:11.921872 30663 rebalancer_tool.cc:1136] tablet 3598dbda342f49aea2384b5beb17d7a1: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:11.964555 30663 rebalancer_tool.cc:1136] tablet a7fd7f8583e14c91a932767ebe23b419: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
I20240502 08:00:11.982380 30663 rebalancer_tool.cc:1136] tablet 6a40382745f84f0eadf9e0ee2e057289: 'c39bcb719db947b8afb0bb141e1f7a4e' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
I20240502 08:00:12.050334 30663 rebalancer_tool.cc:1136] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '' move completed: OK
I20240502 08:00:12.090938 30663 rebalancer_tool.cc:1136] tablet c65fe7f823df4b4996769fc784d3adb7: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: OK
W20240502 08:00:12.609226 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server 6efe17dcca7b4babad4785d19552b699 to server ed3ca7e829bc42b5a8e9a7d1bdcc5a44
I20240502 08:00:12.689064 30663 rebalancer_tool.cc:1074] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: '6c68c916f39a45f697df7eaf8973225a' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:12.784926 30663 rebalancer_tool.cc:1074] tablet 3288b438832e488e8257227cb8934565: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:12.886162 30663 rebalancer_tool.cc:1074] tablet afc9c4d9dd5b4c039686ccd918616859: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'de423605477a4819a9569821f59f03e4' move scheduled
I20240502 08:00:13.051918 30663 rebalancer_tool.cc:1074] tablet 4cb29b8b566747e5b82a8969a070123e: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move scheduled
I20240502 08:00:13.154949 30663 rebalancer_tool.cc:1074] tablet ab785e07fa8248e5889bea8ac755d002: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move scheduled
I20240502 08:00:13.236989 30663 rebalancer_tool.cc:1074] tablet 2ba62cb7ab7b429d85e0ac037c127cb4: '6c68c916f39a45f697df7eaf8973225a' -> '71922d97e1a4414698ecd14a7b2dedec' move scheduled
I20240502 08:00:13.506255 30663 rebalancer_tool.cc:1074] tablet a7fd7f8583e14c91a932767ebe23b419: 'de423605477a4819a9569821f59f03e4' -> 'e5e9553dabc94f218f6427c1685ffc38' move scheduled
I20240502 08:00:13.622416 30663 rebalancer_tool.cc:1074] tablet 21d939e5035541a4b003c5d276493d5c: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move scheduled
I20240502 08:00:13.782668 30663 rebalancer_tool.cc:1074] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'de423605477a4819a9569821f59f03e4' -> '' move scheduled
I20240502 08:00:13.899003 30663 rebalancer_tool.cc:1084] tablet 9c7e7dadff0a4cd08deb043a37a54424: '6c68c916f39a45f697df7eaf8973225a' -> 'c39bcb719db947b8afb0bb141e1f7a4e' move ignored: Illegal state: tablet 9c7e7dadff0a4cd08deb043a37a54424: replica c39bcb719db947b8afb0bb141e1f7a4e is already present
I20240502 08:00:13.899106 30663 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 08:00:14.305410 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table ec76f09dd69640ff9c4a1023602ac6cd: could not find any suitable replica to move from server 6c68c916f39a45f697df7eaf8973225a to server ed3ca7e829bc42b5a8e9a7d1bdcc5a44
W20240502 08:00:14.305507 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server e5e9553dabc94f218f6427c1685ffc38 to server ed3ca7e829bc42b5a8e9a7d1bdcc5a44
W20240502 08:00:14.305567 30663 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f8cc28afdd9143869e64c179746de6ab: could not find any suitable replica to move from server 29126d5aee924f81bdd47cc7153420a1 to server 6c68c916f39a45f697df7eaf8973225a
I20240502 08:00:15.338749 30663 rebalancer_tool.cc:1074] tablet 6a40382745f84f0eadf9e0ee2e057289: 'e5e9553dabc94f218f6427c1685ffc38' -> 'c39bcb719db947b8afb0bb141e1f7a4e' move scheduled
I20240502 08:00:15.374823 30663 rebalancer_tool.cc:1074] tablet 167b5f3b818048af81b1350839ebe489: '6efe17dcca7b4babad4785d19552b699' -> 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' move scheduled
I20240502 08:00:16.240623 30663 rebalancer_tool.cc:1074] tablet 698f494a22d34e659766b97cb24d0293: 'e5e9553dabc94f218f6427c1685ffc38' -> '71922d97e1a4414698ecd14a7b2dedec' move scheduled
I20240502 08:00:17.189056 30663 rebalancer_tool.cc:1074] tablet fddc382ab3164ccdaa86f802c83da8be: 'e5e9553dabc94f218f6427c1685ffc38' -> '71922d97e1a4414698ecd14a7b2dedec' move scheduled
I20240502 08:00:18.482676 30663 rebalancer_tool.cc:1136] tablet 2ba62cb7ab7b429d85e0ac037c127cb4: '6c68c916f39a45f697df7eaf8973225a' -> '71922d97e1a4414698ecd14a7b2dedec' move completed: Incomplete: tablet 2ba62cb7ab7b429d85e0ac037c127cb4, TS 6c68c916f39a45f697df7eaf8973225a -> TS 71922d97e1a4414698ecd14a7b2dedec move failed, target replica disappeared
I20240502 08:00:20.378001 30663 rebalancer_tool.cc:1136] tablet 167b5f3b818048af81b1350839ebe489: '6efe17dcca7b4babad4785d19552b699' -> 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' move completed: OK
I20240502 08:00:22.952569 30663 rebalancer_tool.cc:1136] tablet ab785e07fa8248e5889bea8ac755d002: 'c39bcb719db947b8afb0bb141e1f7a4e' -> '29126d5aee924f81bdd47cc7153420a1' move completed: OK
I20240502 08:00:23.295511 30663 rebalancer_tool.cc:1136] tablet afc9c4d9dd5b4c039686ccd918616859: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> 'de423605477a4819a9569821f59f03e4' move completed: OK
I20240502 08:05:14.190006 30663 rebalancer_tool.cc:1136] tablet a7fd7f8583e14c91a932767ebe23b419: 'de423605477a4819a9569821f59f03e4' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: Incomplete: tablet a7fd7f8583e14c91a932767ebe23b419, TS de423605477a4819a9569821f59f03e4 -> TS e5e9553dabc94f218f6427c1685ffc38 move failed, target replica disappeared
I20240502 08:05:14.908051 30663 rebalancer_tool.cc:1136] tablet 0499a3f3e6bd4069bf992e32f1abfdb0: '6c68c916f39a45f697df7eaf8973225a' -> 'e5e9553dabc94f218f6427c1685ffc38' move completed: Incomplete: tablet 0499a3f3e6bd4069bf992e32f1abfdb0, TS 6c68c916f39a45f697df7eaf8973225a -> TS e5e9553dabc94f218f6427c1685ffc38 move failed, target replica disappeared
I20240502 08:05:14.916718 30663 rebalancer_tool.cc:1136] tablet 6a40382745f84f0eadf9e0ee2e057289: 'e5e9553dabc94f218f6427c1685ffc38' -> 'c39bcb719db947b8afb0bb141e1f7a4e' move completed: OK
I20240502 08:05:16.113672 30663 rebalancer_tool.cc:1136] tablet 21d939e5035541a4b003c5d276493d5c: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:05:16.119262 30663 rebalancer_tool.cc:1136] tablet 9e1cd0ec72d1429f9621d217b1bf845a: 'de423605477a4819a9569821f59f03e4' -> '' move completed: OK
I20240502 08:05:16.125236 30663 rebalancer_tool.cc:1136] tablet 698f494a22d34e659766b97cb24d0293: 'e5e9553dabc94f218f6427c1685ffc38' -> '71922d97e1a4414698ecd14a7b2dedec' move completed: OK
I20240502 08:05:16.262876 30663 rebalancer_tool.cc:1136] tablet 3288b438832e488e8257227cb8934565: '71922d97e1a4414698ecd14a7b2dedec' -> '6efe17dcca7b4babad4785d19552b699' move completed: OK
I20240502 08:05:16.767689 30663 rebalancer_tool.cc:1136] tablet fddc382ab3164ccdaa86f802c83da8be: 'e5e9553dabc94f218f6427c1685ffc38' -> '71922d97e1a4414698ecd14a7b2dedec' move completed: OK
I20240502 08:05:16.782116 30663 rebalancer_tool.cc:1136] tablet 4cb29b8b566747e5b82a8969a070123e: 'ed3ca7e829bc42b5a8e9a7d1bdcc5a44' -> '' move completed: OK
Illegal state: tablet server 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): unacceptable health status UNAVAILABLE
E20240502 08:05:16.929605 16531 external_mini_cluster.cc:943] Process with UUID 6c68c916f39a45f697df7eaf8973225a has crashed
E20240502 08:05:16.929672 16531 external_mini_cluster.cc:943] Process with UUID e5e9553dabc94f218f6427c1685ffc38 has crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/mini-cluster/external_mini_cluster.cc:947: Failure
Expected equality of these values:
0
num_crashes
Which is: 2
At least one process crashed
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/rebalancer_tool-test.cc:1284: Failure
Expected: cluster_->AssertNoCrashes() doesn't generate new fatal failures in the current thread.
Actual: it does.
I20240502 08:05:16.930408 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28317
I20240502 08:05:16.939378 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28448
W20240502 08:05:16.941015 28597 consensus_peers.cc:485] T dcd68a21ed7b4205a2f0cf75f80df0e6 P ed3ca7e829bc42b5a8e9a7d1bdcc5a44 -> Peer 6c68c916f39a45f697df7eaf8973225a (127.16.36.197:35407): Couldn't send request to peer 6c68c916f39a45f697df7eaf8973225a. Status: Network error: Client connection negotiation failed: client connection to 127.16.36.197:35407: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:16.950506 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28579
I20240502 08:05:16.958796 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28710
I20240502 08:05:16.967291 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 30264
I20240502 08:05:16.974601 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 30395
I20240502 08:05:16.982525 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 28226
2024-05-02T08:05:16Z chronyd exiting
I20240502 08:05:17.001467 16531 test_util.cc:175] -----------------------------------------------
I20240502 08:05:17.001533 16531 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.ConcurrentRebalancersTest.TwoConcurrentRebalancers_0.1714636688671911-16531-0
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00> (331434 ms)
[----------] 1 test from ConcurrentRebalancersTest (331434 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0
I20240502 08:05:17.002247 16531 test_util.cc:268] Using random seed: 273093124
I20240502 08:05:17.003010 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:05:17.003046 16531 ts_itest-base.cc:113] --------------
I20240502 08:05:17.003065 16531 ts_itest-base.cc:114] 4 tablet servers
I20240502 08:05:17.003083 16531 ts_itest-base.cc:115] 1 replicas per TS
I20240502 08:05:17.003113 16531 ts_itest-base.cc:116] --------------
2024-05-02T08:05:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:05:17Z Disabled control of system clock
I20240502 08:05:17.011384 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:46305
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:46305
--raft_prepare_replacement_before_eviction=false
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 08:05:17.119917 15892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:17.120155 15892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:17.120195 15892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:17.123322 15892 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:17.123404 15892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:05:17.123428 15892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:17.123447 15892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:05:17.123464 15892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:05:17.127357 15892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:46305
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:46305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.15892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:17.128343 15892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:17.129561 15892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:17.137267 15897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.137672 15898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.138303 15900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:17.139346 15892 server_base.cc:1034] running on GCE node
I20240502 08:05:17.139842 15892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:17.140843 15892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:17.141986 15892 hybrid_clock.cc:648] HybridClock initialized: now 1714637117141966 us; error 37 us; skew 500 ppm
I20240502 08:05:17.143955 15892 webserver.cc:460] Webserver started at http://127.16.36.254:41941/ using document root <none> and password file <none>
I20240502 08:05:17.144595 15892 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:17.144661 15892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:17.144855 15892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:17.146579 15892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.147078 15892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.150619 15892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 08:05:17.153143 15906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.154242 15892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:17.154382 15892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.154482 15892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:17.176074 15892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:17.176833 15892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:17.177009 15892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:17.183759 15892 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:46305
I20240502 08:05:17.183919 15958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:46305 every 8 connection(s)
I20240502 08:05:17.184857 15892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:05:17.186654 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 15892
I20240502 08:05:17.186801 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:05:17.187705 15959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:17.193696 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e: Bootstrap starting.
I20240502 08:05:17.196074 15959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:17.196810 15959 log.cc:824] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:17.198446 15959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e: No bootstrap required, opened a new log
I20240502 08:05:17.200969 15959 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 46305 } }
I20240502 08:05:17.201148 15959 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:17.201187 15959 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec543b338b9f47fcb97c6c0f1cfeae9e, State: Initialized, Role: FOLLOWER
I20240502 08:05:17.201584 15959 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 46305 } }
I20240502 08:05:17.201696 15959 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:17.201733 15959 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:17.201814 15959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:17.202627 15959 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 46305 } }
I20240502 08:05:17.202935 15959 leader_election.cc:302] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec543b338b9f47fcb97c6c0f1cfeae9e; no voters:
I20240502 08:05:17.203200 15959 leader_election.cc:288] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:17.203631 15964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:17.203816 15964 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [term 1 LEADER]: Becoming Leader. State: Replica: ec543b338b9f47fcb97c6c0f1cfeae9e, State: Running, Role: LEADER
I20240502 08:05:17.204121 15964 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 46305 } }
I20240502 08:05:17.204222 15959 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:05:17.205504 15966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec543b338b9f47fcb97c6c0f1cfeae9e. Latest consensus state: current_term: 1 leader_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 46305 } } }
I20240502 08:05:17.205629 15966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:17.205904 15965 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec543b338b9f47fcb97c6c0f1cfeae9e" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 46305 } } }
I20240502 08:05:17.205983 15965 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e [sys.catalog]: This master's current role is: LEADER
W20240502 08:05:17.208839 15979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240502 08:05:17.208918 15979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240502 08:05:17.208979 15975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:05:17.209689 15975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:05:17.214406 15975 catalog_manager.cc:1347] Generated new cluster ID: b9558c50f5b24255b1ae1c430dc5c029
I20240502 08:05:17.214489 15975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:05:17.228693 15975 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:05:17.229831 15975 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:05:17.243530 15975 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ec543b338b9f47fcb97c6c0f1cfeae9e: Generated new TSK 0
I20240502 08:05:17.244251 15975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:05:17.248549 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:05:17.361440 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:17.361699 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:17.361745 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:17.365123 15983 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:17.365206 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:17.365280 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 08:05:17.369170 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--never_fsync=true
--heap_profile_path=/tmp/kudu.15983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:17.370159 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:17.371321 15983 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:17.380683 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.381218 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.381667 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:17.382700 15983 server_base.cc:1034] running on GCE node
I20240502 08:05:17.383162 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:17.383920 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:17.385109 15983 hybrid_clock.cc:648] HybridClock initialized: now 1714637117385098 us; error 36 us; skew 500 ppm
I20240502 08:05:17.387125 15983 webserver.cc:460] Webserver started at http://127.16.36.193:35945/ using document root <none> and password file <none>
I20240502 08:05:17.387714 15983 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:17.387782 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:17.387984 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:17.389771 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e92df7bd75144c989f39ece64a410221"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.390272 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e92df7bd75144c989f39ece64a410221"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.393731 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:05:17.396064 15997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.397163 15983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:17.397284 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e92df7bd75144c989f39ece64a410221"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.397377 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:17.411687 15983 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:17.412333 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:17.412564 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:17.413108 15983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:17.414043 15983 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:17.414093 15983 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.414160 15983 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:17.414186 15983 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.422814 15983 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:41969
I20240502 08:05:17.422840 16110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:41969 every 8 connection(s)
I20240502 08:05:17.423794 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:05:17.424697 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 15983
I20240502 08:05:17.424829 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:05:17.427093 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:05:17.434352 16111 heartbeater.cc:347] Connected to a master server at 127.16.36.254:46305
I20240502 08:05:17.434662 16111 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:17.435319 16111 heartbeater.cc:511] Master 127.16.36.254:46305 requested a full tablet report, sending...
I20240502 08:05:17.437023 15923 ts_manager.cc:194] Registered new tserver with Master: e92df7bd75144c989f39ece64a410221 (127.16.36.193:41969)
I20240502 08:05:17.438202 15923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:49261
W20240502 08:05:17.540912 16114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:17.541183 16114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:17.541265 16114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:17.544364 16114 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:17.544497 16114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:17.544641 16114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 08:05:17.548393 16114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--never_fsync=true
--heap_profile_path=/tmp/kudu.16114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:17.549404 16114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:17.550550 16114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:17.559360 16122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.559921 16119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.560420 16120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:17.561367 16114 server_base.cc:1034] running on GCE node
I20240502 08:05:17.561792 16114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:17.562462 16114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:17.563627 16114 hybrid_clock.cc:648] HybridClock initialized: now 1714637117563612 us; error 29 us; skew 500 ppm
I20240502 08:05:17.565502 16114 webserver.cc:460] Webserver started at http://127.16.36.194:39111/ using document root <none> and password file <none>
I20240502 08:05:17.566041 16114 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:17.566093 16114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:17.566260 16114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:17.567932 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4669623dcbe9447ab43d30b1c7713228"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.568387 16114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4669623dcbe9447ab43d30b1c7713228"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.571878 16114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:17.574362 16128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.575404 16114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:17.575521 16114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4669623dcbe9447ab43d30b1c7713228"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.575623 16114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:17.598037 16114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:17.598702 16114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:17.598872 16114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:17.599428 16114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:17.600339 16114 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:17.600394 16114 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.600471 16114 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:17.600497 16114 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.609493 16114 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:38567
I20240502 08:05:17.609522 16242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:38567 every 8 connection(s)
I20240502 08:05:17.610474 16114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:05:17.613880 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16114
I20240502 08:05:17.614015 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:05:17.616410 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:05:17.619976 16243 heartbeater.cc:347] Connected to a master server at 127.16.36.254:46305
I20240502 08:05:17.620230 16243 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:17.620718 16243 heartbeater.cc:511] Master 127.16.36.254:46305 requested a full tablet report, sending...
I20240502 08:05:17.621723 15923 ts_manager.cc:194] Registered new tserver with Master: 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194:38567)
I20240502 08:05:17.622282 15923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:46439
W20240502 08:05:17.728415 16246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:17.728724 16246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:17.728797 16246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:17.731834 16246 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:17.731909 16246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:17.732014 16246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 08:05:17.735576 16246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--never_fsync=true
--heap_profile_path=/tmp/kudu.16246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:17.736532 16246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:17.737669 16246 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:17.746588 16254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.747054 16251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.747658 16252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:17.748430 16246 server_base.cc:1034] running on GCE node
I20240502 08:05:17.748903 16246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:17.749579 16246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:17.750739 16246 hybrid_clock.cc:648] HybridClock initialized: now 1714637117750725 us; error 29 us; skew 500 ppm
I20240502 08:05:17.752736 16246 webserver.cc:460] Webserver started at http://127.16.36.195:34695/ using document root <none> and password file <none>
I20240502 08:05:17.753291 16246 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:17.753346 16246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:17.753520 16246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:17.755264 16246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "62c874074d234b159322f057cbc4d217"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.755740 16246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "62c874074d234b159322f057cbc4d217"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.759488 16246 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 08:05:17.762243 16260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.763377 16246 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 08:05:17.763499 16246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "62c874074d234b159322f057cbc4d217"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.763605 16246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:17.775985 16246 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:17.776674 16246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:17.776844 16246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:17.777405 16246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:17.778326 16246 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:17.778378 16246 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.778442 16246 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:17.778470 16246 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.787065 16246 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:36037
I20240502 08:05:17.787086 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:36037 every 8 connection(s)
I20240502 08:05:17.788072 16246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:05:17.793107 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16246
I20240502 08:05:17.793246 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:05:17.795549 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:05:17.797206 16374 heartbeater.cc:347] Connected to a master server at 127.16.36.254:46305
I20240502 08:05:17.797446 16374 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:17.797900 16374 heartbeater.cc:511] Master 127.16.36.254:46305 requested a full tablet report, sending...
I20240502 08:05:17.798890 15923 ts_manager.cc:194] Registered new tserver with Master: 62c874074d234b159322f057cbc4d217 (127.16.36.195:36037)
I20240502 08:05:17.799666 15923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:51259
W20240502 08:05:17.911724 16377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:17.912022 16377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:17.912075 16377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:17.915629 16377 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:17.915724 16377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:17.915802 16377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 08:05:17.919658 16377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:46305
--never_fsync=true
--heap_profile_path=/tmp/kudu.16377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:17.920646 16377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:17.921772 16377 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:17.930761 16382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:17.931317 16383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:17.932245 16377 server_base.cc:1034] running on GCE node
W20240502 08:05:17.932690 16385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:17.933187 16377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:17.933918 16377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:17.935107 16377 hybrid_clock.cc:648] HybridClock initialized: now 1714637117935091 us; error 33 us; skew 500 ppm
I20240502 08:05:17.937078 16377 webserver.cc:460] Webserver started at http://127.16.36.196:36905/ using document root <none> and password file <none>
I20240502 08:05:17.937625 16377 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:17.937680 16377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:17.937853 16377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:17.939551 16377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "dce9d531641c45abb70201c9fa6b66ca"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.940013 16377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "dce9d531641c45abb70201c9fa6b66ca"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.943821 16377 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 08:05:17.947067 16391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.948436 16377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:17.948629 16377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dce9d531641c45abb70201c9fa6b66ca"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:17.948747 16377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:17.978397 16377 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:17.979027 16377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:17.979192 16377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:17.979796 16377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:17.980767 16377 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:17.980818 16377 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.980885 16377 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:17.980911 16377 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:17.990310 16377 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:33465
I20240502 08:05:17.990379 16504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:33465 every 8 connection(s)
I20240502 08:05:17.991294 16377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:05:17.993446 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16377
I20240502 08:05:17.993604 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:05:18.000327 16505 heartbeater.cc:347] Connected to a master server at 127.16.36.254:46305
I20240502 08:05:18.000576 16505 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:18.001081 16505 heartbeater.cc:511] Master 127.16.36.254:46305 requested a full tablet report, sending...
I20240502 08:05:18.002048 15923 ts_manager.cc:194] Registered new tserver with Master: dce9d531641c45abb70201c9fa6b66ca (127.16.36.196:33465)
I20240502 08:05:18.002628 15923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:49501
I20240502 08:05:18.007846 16531 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 08:05:18.016674 15923 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39472:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 08:05:18.033211 16308 tablet_service.cc:1467] Processing CreateTablet for tablet c4a48d1332154a1ba61019567823d00e (DEFAULT_TABLE table=TestTable [id=4d007c2138ad49ef94db65444ff5b025]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:18.034116 16308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4a48d1332154a1ba61019567823d00e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:18.039273 16519 tablet_bootstrap.cc:492] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Bootstrap starting.
I20240502 08:05:18.041205 16519 tablet_bootstrap.cc:654] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:18.041893 16519 log.cc:824] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:18.043525 16519 tablet_bootstrap.cc:492] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: No bootstrap required, opened a new log
I20240502 08:05:18.043704 16519 ts_tablet_manager.cc:1367] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 08:05:18.046645 16519 raft_consensus.cc:348] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:18.046840 16519 raft_consensus.cc:374] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:18.046893 16519 raft_consensus.cc:729] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c874074d234b159322f057cbc4d217, State: Initialized, Role: FOLLOWER
I20240502 08:05:18.047312 16519 consensus_queue.cc:260] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:18.047454 16519 raft_consensus.cc:388] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:18.047536 16519 raft_consensus.cc:482] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:18.047643 16519 raft_consensus.cc:3037] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:18.048538 16519 raft_consensus.cc:504] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:18.048864 16519 leader_election.cc:302] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62c874074d234b159322f057cbc4d217; no voters:
I20240502 08:05:18.049175 16519 leader_election.cc:288] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:18.049307 16521 raft_consensus.cc:2781] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:18.049530 16521 raft_consensus.cc:686] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 LEADER]: Becoming Leader. State: Replica: 62c874074d234b159322f057cbc4d217, State: Running, Role: LEADER
I20240502 08:05:18.049886 16374 heartbeater.cc:503] Master 127.16.36.254:46305 was elected leader, sending a full tablet report...
I20240502 08:05:18.049849 16521 consensus_queue.cc:237] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:18.050060 16519 ts_tablet_manager.cc:1398] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 08:05:18.053236 15923 catalog_manager.cc:5484] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 reported cstate change: term changed from 0 to 1, leader changed from <none> to 62c874074d234b159322f057cbc4d217 (127.16.36.195). New cstate: current_term: 1 leader_uuid: "62c874074d234b159322f057cbc4d217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } } }
I20240502 08:05:18.060217 16531 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 08:05:18.061566 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e92df7bd75144c989f39ece64a410221 to finish bootstrapping
I20240502 08:05:18.067709 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4669623dcbe9447ab43d30b1c7713228 to finish bootstrapping
I20240502 08:05:18.074131 16531 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 62c874074d234b159322f057cbc4d217 to finish bootstrapping
I20240502 08:05:18.079110 16531 ts_itest-base.cc:233] Waiting for 0 tablets on tserver dce9d531641c45abb70201c9fa6b66ca to finish bootstrapping
I20240502 08:05:18.084949 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16246
I20240502 08:05:18.091073 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16377
I20240502 08:05:18.441136 16111 heartbeater.cc:503] Master 127.16.36.254:46305 was elected leader, sending a full tablet report...
I20240502 08:05:18.624277 16243 heartbeater.cc:503] Master 127.16.36.254:46305 was elected leader, sending a full tablet report...
I20240502 08:05:21.848117 15923 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39472:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:05:21.860114 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 3e0f7c4d3b8f432399ebd5bf9e81a59a (DEFAULT_TABLE table=rebalance_test_table_0 [id=bb2a8dd84d3d48c48908422ff3a2ba8d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:21.860114 16044 tablet_service.cc:1467] Processing CreateTablet for tablet c0a02f15e22842a1b46060531feb0312 (DEFAULT_TABLE table=rebalance_test_table_0 [id=bb2a8dd84d3d48c48908422ff3a2ba8d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:21.860126 16045 tablet_service.cc:1467] Processing CreateTablet for tablet d5145f37eceb4bca96b2d0d7bd1e842e (DEFAULT_TABLE table=rebalance_test_table_0 [id=bb2a8dd84d3d48c48908422ff3a2ba8d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:21.861115 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5145f37eceb4bca96b2d0d7bd1e842e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:21.861128 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e0f7c4d3b8f432399ebd5bf9e81a59a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:21.861462 16044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0a02f15e22842a1b46060531feb0312. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:21.866357 16543 tablet_bootstrap.cc:492] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228: Bootstrap starting.
I20240502 08:05:21.866731 16544 tablet_bootstrap.cc:492] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221: Bootstrap starting.
I20240502 08:05:21.868439 16543 tablet_bootstrap.cc:654] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:21.868479 16544 tablet_bootstrap.cc:654] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:21.869110 16543 log.cc:824] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:21.869110 16544 log.cc:824] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:21.870559 16544 tablet_bootstrap.cc:492] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221: No bootstrap required, opened a new log
I20240502 08:05:21.870568 16543 tablet_bootstrap.cc:492] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228: No bootstrap required, opened a new log
I20240502 08:05:21.870724 16543 ts_tablet_manager.cc:1367] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 08:05:21.870724 16544 ts_tablet_manager.cc:1367] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 08:05:21.873833 16544 raft_consensus.cc:348] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.873833 16543 raft_consensus.cc:348] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:21.874012 16543 raft_consensus.cc:374] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:21.874018 16544 raft_consensus.cc:374] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:21.874045 16543 raft_consensus.cc:729] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Initialized, Role: FOLLOWER
I20240502 08:05:21.874070 16544 raft_consensus.cc:729] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92df7bd75144c989f39ece64a410221, State: Initialized, Role: FOLLOWER
I20240502 08:05:21.874446 16544 consensus_queue.cc:260] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.874447 16543 consensus_queue.cc:260] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:21.874562 16543 raft_consensus.cc:388] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:21.874567 16544 raft_consensus.cc:388] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:21.874626 16543 raft_consensus.cc:482] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:21.874629 16544 raft_consensus.cc:482] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:21.874732 16544 raft_consensus.cc:3037] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:21.874733 16543 raft_consensus.cc:3037] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:21.875501 16544 raft_consensus.cc:504] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.875517 16543 raft_consensus.cc:504] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:21.875805 16543 leader_election.cc:302] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669623dcbe9447ab43d30b1c7713228; no voters:
I20240502 08:05:21.875808 16544 leader_election.cc:302] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92df7bd75144c989f39ece64a410221; no voters:
I20240502 08:05:21.876099 16544 leader_election.cc:288] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:21.876099 16543 leader_election.cc:288] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:21.876188 16547 raft_consensus.cc:2781] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:21.876189 16548 raft_consensus.cc:2781] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:21.876368 16548 raft_consensus.cc:686] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [term 1 LEADER]: Becoming Leader. State: Replica: e92df7bd75144c989f39ece64a410221, State: Running, Role: LEADER
I20240502 08:05:21.876513 16547 raft_consensus.cc:686] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [term 1 LEADER]: Becoming Leader. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Running, Role: LEADER
I20240502 08:05:21.876686 16548 consensus_queue.cc:237] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.876806 16547 consensus_queue.cc:237] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:21.876986 16544 ts_tablet_manager.cc:1398] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20240502 08:05:21.876993 16543 ts_tablet_manager.cc:1398] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20240502 08:05:21.877620 16544 tablet_bootstrap.cc:492] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221: Bootstrap starting.
I20240502 08:05:21.879267 15923 catalog_manager.cc:5484] T c0a02f15e22842a1b46060531feb0312 P e92df7bd75144c989f39ece64a410221 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92df7bd75144c989f39ece64a410221 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "e92df7bd75144c989f39ece64a410221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } } }
I20240502 08:05:21.879899 16544 tablet_bootstrap.cc:654] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:21.880175 15922 catalog_manager.cc:5484] T 3e0f7c4d3b8f432399ebd5bf9e81a59a P 4669623dcbe9447ab43d30b1c7713228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "4669623dcbe9447ab43d30b1c7713228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } } }
I20240502 08:05:21.881008 16544 tablet_bootstrap.cc:492] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221: No bootstrap required, opened a new log
I20240502 08:05:21.881096 16544 ts_tablet_manager.cc:1367] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:05:21.881613 16544 raft_consensus.cc:348] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.881744 16544 raft_consensus.cc:374] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:21.881819 16544 raft_consensus.cc:729] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92df7bd75144c989f39ece64a410221, State: Initialized, Role: FOLLOWER
I20240502 08:05:21.881985 16544 consensus_queue.cc:260] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.882097 16544 raft_consensus.cc:388] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:21.882177 16544 raft_consensus.cc:482] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:21.882242 16544 raft_consensus.cc:3037] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:21.883153 16544 raft_consensus.cc:504] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.883284 16544 leader_election.cc:302] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92df7bd75144c989f39ece64a410221; no voters:
I20240502 08:05:21.883378 16544 leader_election.cc:288] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:21.883435 16551 raft_consensus.cc:2781] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:21.883539 16551 raft_consensus.cc:686] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [term 1 LEADER]: Becoming Leader. State: Replica: e92df7bd75144c989f39ece64a410221, State: Running, Role: LEADER
I20240502 08:05:21.883651 16544 ts_tablet_manager.cc:1398] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:05:21.883646 16551 consensus_queue.cc:237] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:21.885733 15922 catalog_manager.cc:5484] T d5145f37eceb4bca96b2d0d7bd1e842e P e92df7bd75144c989f39ece64a410221 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92df7bd75144c989f39ece64a410221 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "e92df7bd75144c989f39ece64a410221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } } }
I20240502 08:05:22.073107 15922 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39472:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:05:22.078353 16177 tablet_service.cc:1467] Processing CreateTablet for tablet c3afa575296949f7b139ffe2a2f53a9c (DEFAULT_TABLE table=rebalance_test_table_1 [id=eecdb390deb64bf3a0a9c1add576efaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.078372 16176 tablet_service.cc:1467] Processing CreateTablet for tablet 0cba4b7f334f4687a32d43b2442d5609 (DEFAULT_TABLE table=rebalance_test_table_1 [id=eecdb390deb64bf3a0a9c1add576efaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.078514 16045 tablet_service.cc:1467] Processing CreateTablet for tablet e66666a128de46239c8bd33b13ef22ff (DEFAULT_TABLE table=rebalance_test_table_1 [id=eecdb390deb64bf3a0a9c1add576efaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.078725 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3afa575296949f7b139ffe2a2f53a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.078802 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e66666a128de46239c8bd33b13ef22ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.078836 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cba4b7f334f4687a32d43b2442d5609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.080718 16544 tablet_bootstrap.cc:492] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221: Bootstrap starting.
I20240502 08:05:22.080940 16543 tablet_bootstrap.cc:492] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228: Bootstrap starting.
I20240502 08:05:22.081739 16544 tablet_bootstrap.cc:654] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.081825 16543 tablet_bootstrap.cc:654] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.082887 16544 tablet_bootstrap.cc:492] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221: No bootstrap required, opened a new log
I20240502 08:05:22.082934 16543 tablet_bootstrap.cc:492] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228: No bootstrap required, opened a new log
I20240502 08:05:22.082973 16544 ts_tablet_manager.cc:1367] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.083016 16543 ts_tablet_manager.cc:1367] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:22.083457 16544 raft_consensus.cc:348] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.083564 16544 raft_consensus.cc:374] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.083602 16544 raft_consensus.cc:729] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92df7bd75144c989f39ece64a410221, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.083689 16543 raft_consensus.cc:348] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.083735 16544 consensus_queue.cc:260] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.083796 16543 raft_consensus.cc:374] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.083815 16544 raft_consensus.cc:388] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.083833 16543 raft_consensus.cc:729] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.083847 16544 raft_consensus.cc:482] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.083895 16544 raft_consensus.cc:3037] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.083969 16543 consensus_queue.cc:260] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.084065 16543 raft_consensus.cc:388] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.084113 16543 raft_consensus.cc:482] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.084160 16543 raft_consensus.cc:3037] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.084760 16544 raft_consensus.cc:504] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.084892 16544 leader_election.cc:302] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92df7bd75144c989f39ece64a410221; no voters:
I20240502 08:05:22.084903 16543 raft_consensus.cc:504] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.084992 16544 leader_election.cc:288] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.085009 16543 leader_election.cc:302] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669623dcbe9447ab43d30b1c7713228; no voters:
I20240502 08:05:22.085067 16548 raft_consensus.cc:2781] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.085103 16543 leader_election.cc:288] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.085165 16548 raft_consensus.cc:686] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [term 1 LEADER]: Becoming Leader. State: Replica: e92df7bd75144c989f39ece64a410221, State: Running, Role: LEADER
I20240502 08:05:22.085171 16549 raft_consensus.cc:2781] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.085290 16548 consensus_queue.cc:237] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.085430 16544 ts_tablet_manager.cc:1398] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:22.085548 16549 raft_consensus.cc:686] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [term 1 LEADER]: Becoming Leader. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Running, Role: LEADER
I20240502 08:05:22.085702 16549 consensus_queue.cc:237] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.085973 16543 ts_tablet_manager.cc:1398] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:05:22.086139 16543 tablet_bootstrap.cc:492] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228: Bootstrap starting.
I20240502 08:05:22.086620 15922 catalog_manager.cc:5484] T e66666a128de46239c8bd33b13ef22ff P e92df7bd75144c989f39ece64a410221 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92df7bd75144c989f39ece64a410221 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "e92df7bd75144c989f39ece64a410221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } } }
I20240502 08:05:22.087075 16543 tablet_bootstrap.cc:654] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.087123 15923 catalog_manager.cc:5484] T c3afa575296949f7b139ffe2a2f53a9c P 4669623dcbe9447ab43d30b1c7713228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "4669623dcbe9447ab43d30b1c7713228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } } }
I20240502 08:05:22.088979 16543 tablet_bootstrap.cc:492] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228: No bootstrap required, opened a new log
I20240502 08:05:22.089062 16543 ts_tablet_manager.cc:1367] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:22.089548 16543 raft_consensus.cc:348] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.089648 16543 raft_consensus.cc:374] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.089682 16543 raft_consensus.cc:729] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.089808 16543 consensus_queue.cc:260] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.089885 16543 raft_consensus.cc:388] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.089932 16543 raft_consensus.cc:482] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.089982 16543 raft_consensus.cc:3037] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.090667 16543 raft_consensus.cc:504] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.090801 16543 leader_election.cc:302] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669623dcbe9447ab43d30b1c7713228; no voters:
I20240502 08:05:22.090893 16543 leader_election.cc:288] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.090977 16552 raft_consensus.cc:2781] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.091068 16552 raft_consensus.cc:686] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [term 1 LEADER]: Becoming Leader. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Running, Role: LEADER
I20240502 08:05:22.091135 16543 ts_tablet_manager.cc:1398] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.091177 16552 consensus_queue.cc:237] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.092324 15923 catalog_manager.cc:5484] T 0cba4b7f334f4687a32d43b2442d5609 P 4669623dcbe9447ab43d30b1c7713228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "4669623dcbe9447ab43d30b1c7713228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } } }
I20240502 08:05:22.263760 15923 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39472:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:05:22.269486 16045 tablet_service.cc:1467] Processing CreateTablet for tablet 3a226ca918c64c169624cf6dfb283b23 (DEFAULT_TABLE table=rebalance_test_table_2 [id=eaa3e0a53cf04b818cb0cebd86a5fd38]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.269534 16176 tablet_service.cc:1467] Processing CreateTablet for tablet f2ba3bf250534204839874d30fdb3d08 (DEFAULT_TABLE table=rebalance_test_table_2 [id=eaa3e0a53cf04b818cb0cebd86a5fd38]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.269575 16044 tablet_service.cc:1467] Processing CreateTablet for tablet a7b5fa8de1fd4977bf31eb7231da4cea (DEFAULT_TABLE table=rebalance_test_table_2 [id=eaa3e0a53cf04b818cb0cebd86a5fd38]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.269820 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a226ca918c64c169624cf6dfb283b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.269820 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2ba3bf250534204839874d30fdb3d08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.270007 16044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7b5fa8de1fd4977bf31eb7231da4cea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.271772 16543 tablet_bootstrap.cc:492] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228: Bootstrap starting.
I20240502 08:05:22.272037 16544 tablet_bootstrap.cc:492] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221: Bootstrap starting.
I20240502 08:05:22.272871 16543 tablet_bootstrap.cc:654] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.273020 16544 tablet_bootstrap.cc:654] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.274103 16543 tablet_bootstrap.cc:492] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228: No bootstrap required, opened a new log
I20240502 08:05:22.274103 16544 tablet_bootstrap.cc:492] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221: No bootstrap required, opened a new log
I20240502 08:05:22.274192 16544 ts_tablet_manager.cc:1367] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:05:22.274192 16543 ts_tablet_manager.cc:1367] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.274657 16543 raft_consensus.cc:348] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.274657 16544 raft_consensus.cc:348] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.274770 16544 raft_consensus.cc:374] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.274767 16543 raft_consensus.cc:374] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.274803 16544 raft_consensus.cc:729] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92df7bd75144c989f39ece64a410221, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.274803 16543 raft_consensus.cc:729] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.274937 16543 consensus_queue.cc:260] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.274938 16544 consensus_queue.cc:260] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.275020 16544 raft_consensus.cc:388] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.275029 16543 raft_consensus.cc:388] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.275068 16543 raft_consensus.cc:482] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.275068 16544 raft_consensus.cc:482] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.275120 16544 raft_consensus.cc:3037] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.275120 16543 raft_consensus.cc:3037] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.275960 16544 raft_consensus.cc:504] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.275960 16543 raft_consensus.cc:504] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.276098 16544 leader_election.cc:302] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92df7bd75144c989f39ece64a410221; no voters:
I20240502 08:05:22.276098 16543 leader_election.cc:302] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669623dcbe9447ab43d30b1c7713228; no voters:
I20240502 08:05:22.276190 16543 leader_election.cc:288] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.276238 16548 raft_consensus.cc:2781] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.276259 16547 raft_consensus.cc:2781] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.276327 16548 raft_consensus.cc:686] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [term 1 LEADER]: Becoming Leader. State: Replica: e92df7bd75144c989f39ece64a410221, State: Running, Role: LEADER
I20240502 08:05:22.276357 16547 raft_consensus.cc:686] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [term 1 LEADER]: Becoming Leader. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Running, Role: LEADER
I20240502 08:05:22.276428 16543 ts_tablet_manager.cc:1398] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 08:05:22.276497 16548 consensus_queue.cc:237] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.276602 16547 consensus_queue.cc:237] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.276888 16544 leader_election.cc:288] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.277420 16544 ts_tablet_manager.cc:1398] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 08:05:22.277599 16544 tablet_bootstrap.cc:492] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221: Bootstrap starting.
I20240502 08:05:22.278015 15923 catalog_manager.cc:5484] T a7b5fa8de1fd4977bf31eb7231da4cea P e92df7bd75144c989f39ece64a410221 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92df7bd75144c989f39ece64a410221 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "e92df7bd75144c989f39ece64a410221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } } }
I20240502 08:05:22.278627 16544 tablet_bootstrap.cc:654] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.279034 15922 catalog_manager.cc:5484] T f2ba3bf250534204839874d30fdb3d08 P 4669623dcbe9447ab43d30b1c7713228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "4669623dcbe9447ab43d30b1c7713228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } } }
I20240502 08:05:22.279837 16544 tablet_bootstrap.cc:492] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221: No bootstrap required, opened a new log
I20240502 08:05:22.279923 16544 ts_tablet_manager.cc:1367] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:05:22.280377 16544 raft_consensus.cc:348] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.280512 16544 raft_consensus.cc:374] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.280555 16544 raft_consensus.cc:729] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92df7bd75144c989f39ece64a410221, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.280678 16544 consensus_queue.cc:260] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.280761 16544 raft_consensus.cc:388] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.280799 16544 raft_consensus.cc:482] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.280840 16544 raft_consensus.cc:3037] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.281725 16544 raft_consensus.cc:504] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.281852 16544 leader_election.cc:302] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92df7bd75144c989f39ece64a410221; no voters:
I20240502 08:05:22.281939 16544 leader_election.cc:288] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.282079 16551 raft_consensus.cc:2781] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.282155 16544 ts_tablet_manager.cc:1398] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.282169 16551 raft_consensus.cc:686] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [term 1 LEADER]: Becoming Leader. State: Replica: e92df7bd75144c989f39ece64a410221, State: Running, Role: LEADER
I20240502 08:05:22.282500 16551 consensus_queue.cc:237] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.284121 15923 catalog_manager.cc:5484] T 3a226ca918c64c169624cf6dfb283b23 P e92df7bd75144c989f39ece64a410221 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92df7bd75144c989f39ece64a410221 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "e92df7bd75144c989f39ece64a410221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } } }
I20240502 08:05:22.457412 15923 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39472:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 08:05:22.462743 16176 tablet_service.cc:1467] Processing CreateTablet for tablet dadd09184ab54e1bb3caf0ce781553a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5bcbb3095d574c988b29008e04bdffd7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.462797 16177 tablet_service.cc:1467] Processing CreateTablet for tablet 98f0148fa9be40e58ecfb4b330975a6d (DEFAULT_TABLE table=rebalance_test_table_3 [id=5bcbb3095d574c988b29008e04bdffd7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.462852 16045 tablet_service.cc:1467] Processing CreateTablet for tablet a30076a542cc4fa88d97991a9bd107ec (DEFAULT_TABLE table=rebalance_test_table_3 [id=5bcbb3095d574c988b29008e04bdffd7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:22.463095 16176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dadd09184ab54e1bb3caf0ce781553a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.463135 16045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30076a542cc4fa88d97991a9bd107ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.464864 16544 tablet_bootstrap.cc:492] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221: Bootstrap starting.
I20240502 08:05:22.464915 16177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98f0148fa9be40e58ecfb4b330975a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:22.464934 16543 tablet_bootstrap.cc:492] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228: Bootstrap starting.
I20240502 08:05:22.465744 16543 tablet_bootstrap.cc:654] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.465759 16544 tablet_bootstrap.cc:654] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.466745 16543 tablet_bootstrap.cc:492] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228: No bootstrap required, opened a new log
I20240502 08:05:22.466835 16543 ts_tablet_manager.cc:1367] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.467183 16544 tablet_bootstrap.cc:492] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221: No bootstrap required, opened a new log
I20240502 08:05:22.467186 16543 raft_consensus.cc:348] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.467238 16544 ts_tablet_manager.cc:1367] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.467257 16543 raft_consensus.cc:374] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.467278 16543 raft_consensus.cc:729] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.467383 16543 consensus_queue.cc:260] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.467445 16543 raft_consensus.cc:388] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.467468 16543 raft_consensus.cc:482] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.467495 16543 raft_consensus.cc:3037] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.467561 16544 raft_consensus.cc:348] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.467626 16544 raft_consensus.cc:374] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.467648 16544 raft_consensus.cc:729] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e92df7bd75144c989f39ece64a410221, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.467737 16544 consensus_queue.cc:260] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.467785 16544 raft_consensus.cc:388] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.467805 16544 raft_consensus.cc:482] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.467831 16544 raft_consensus.cc:3037] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.468114 16543 raft_consensus.cc:504] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.468214 16543 leader_election.cc:302] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669623dcbe9447ab43d30b1c7713228; no voters:
I20240502 08:05:22.468288 16543 leader_election.cc:288] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.468349 16549 raft_consensus.cc:2781] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.468420 16549 raft_consensus.cc:686] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [term 1 LEADER]: Becoming Leader. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Running, Role: LEADER
I20240502 08:05:22.468561 16543 ts_tablet_manager.cc:1398] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.468544 16549 consensus_queue.cc:237] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.468691 16543 tablet_bootstrap.cc:492] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228: Bootstrap starting.
I20240502 08:05:22.468956 16544 raft_consensus.cc:504] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.469094 16544 leader_election.cc:302] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e92df7bd75144c989f39ece64a410221; no voters:
I20240502 08:05:22.469226 16551 raft_consensus.cc:2781] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.469300 16551 raft_consensus.cc:686] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [term 1 LEADER]: Becoming Leader. State: Replica: e92df7bd75144c989f39ece64a410221, State: Running, Role: LEADER
I20240502 08:05:22.469422 16551 consensus_queue.cc:237] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } }
I20240502 08:05:22.469826 16543 tablet_bootstrap.cc:654] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:22.470021 16544 leader_election.cc:288] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.470283 16544 ts_tablet_manager.cc:1398] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:22.470952 16543 tablet_bootstrap.cc:492] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228: No bootstrap required, opened a new log
I20240502 08:05:22.471043 16543 ts_tablet_manager.cc:1367] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.470994 15923 catalog_manager.cc:5484] T dadd09184ab54e1bb3caf0ce781553a1 P 4669623dcbe9447ab43d30b1c7713228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "4669623dcbe9447ab43d30b1c7713228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } } }
I20240502 08:05:22.471542 16543 raft_consensus.cc:348] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.471625 16543 raft_consensus.cc:374] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:22.471654 16543 raft_consensus.cc:729] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.471591 15922 catalog_manager.cc:5484] T a30076a542cc4fa88d97991a9bd107ec P e92df7bd75144c989f39ece64a410221 reported cstate change: term changed from 0 to 1, leader changed from <none> to e92df7bd75144c989f39ece64a410221 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "e92df7bd75144c989f39ece64a410221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e92df7bd75144c989f39ece64a410221" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 41969 } } }
I20240502 08:05:22.471758 16543 consensus_queue.cc:260] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.471825 16543 raft_consensus.cc:388] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.471858 16543 raft_consensus.cc:482] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.471895 16543 raft_consensus.cc:3037] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:22.472755 16543 raft_consensus.cc:504] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.472887 16543 leader_election.cc:302] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4669623dcbe9447ab43d30b1c7713228; no voters:
I20240502 08:05:22.472992 16543 leader_election.cc:288] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:22.473111 16547 raft_consensus.cc:2781] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:22.473209 16547 raft_consensus.cc:686] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [term 1 LEADER]: Becoming Leader. State: Replica: 4669623dcbe9447ab43d30b1c7713228, State: Running, Role: LEADER
I20240502 08:05:22.473330 16547 consensus_queue.cc:237] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } }
I20240502 08:05:22.473206 16543 ts_tablet_manager.cc:1398] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:05:22.475057 15922 catalog_manager.cc:5484] T 98f0148fa9be40e58ecfb4b330975a6d P 4669623dcbe9447ab43d30b1c7713228 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4669623dcbe9447ab43d30b1c7713228 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "4669623dcbe9447ab43d30b1c7713228" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4669623dcbe9447ab43d30b1c7713228" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 38567 } } }
I20240502 08:05:22.648247 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:36037
--local_ip_for_outbound_sockets=127.16.36.195
--tserver_master_addrs=127.16.36.254:46305
--webserver_port=34695
--webserver_interface=127.16.36.195
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
W20240502 08:05:22.756983 16650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:22.757239 16650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:22.757287 16650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:22.760497 16650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:22.760577 16650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:22.760651 16650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 08:05:22.764510 16650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:36037
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=34695
--tserver_master_addrs=127.16.36.254:46305
--never_fsync=true
--heap_profile_path=/tmp/kudu.16650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:22.765403 16650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:22.766484 16650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:22.775184 16655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.775527 16656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.776152 16658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:22.777115 16650 server_base.cc:1034] running on GCE node
I20240502 08:05:22.777560 16650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:22.778297 16650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:22.779479 16650 hybrid_clock.cc:648] HybridClock initialized: now 1714637122779450 us; error 41 us; skew 500 ppm
I20240502 08:05:22.781505 16650 webserver.cc:460] Webserver started at http://127.16.36.195:34695/ using document root <none> and password file <none>
I20240502 08:05:22.782071 16650 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:22.782153 16650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:22.785601 16650 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:22.787569 16664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.788733 16650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:22.788852 16650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "62c874074d234b159322f057cbc4d217"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:22.789265 16650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:22.803772 16650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:22.804435 16650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:22.804625 16650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:22.805168 16650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:22.806319 16671 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240502 08:05:22.809408 16650 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 08:05:22.809465 16650 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240502 08:05:22.809532 16650 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 08:05:22.810978 16650 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 08:05:22.811022 16650 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240502 08:05:22.811275 16671 tablet_bootstrap.cc:492] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Bootstrap starting.
I20240502 08:05:22.821833 16650 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:36037
I20240502 08:05:22.821852 16778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:36037 every 8 connection(s)
I20240502 08:05:22.823173 16650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:05:22.824421 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16650
I20240502 08:05:22.824954 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:33465
--local_ip_for_outbound_sockets=127.16.36.196
--tserver_master_addrs=127.16.36.254:46305
--webserver_port=36905
--webserver_interface=127.16.36.196
--builtin_ntp_servers=127.16.36.212:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false with env {}
I20240502 08:05:22.828192 16671 log.cc:824] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:22.831854 16671 tablet_bootstrap.cc:492] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:22.832334 16671 tablet_bootstrap.cc:492] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Bootstrap complete.
I20240502 08:05:22.833890 16671 ts_tablet_manager.cc:1367] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.003s
I20240502 08:05:22.834085 16779 heartbeater.cc:347] Connected to a master server at 127.16.36.254:46305
I20240502 08:05:22.834304 16779 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:22.834755 16779 heartbeater.cc:511] Master 127.16.36.254:46305 requested a full tablet report, sending...
I20240502 08:05:22.836407 15922 ts_manager.cc:194] Re-registered known tserver with Master: 62c874074d234b159322f057cbc4d217 (127.16.36.195:36037)
I20240502 08:05:22.837204 15922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:35171
I20240502 08:05:22.837783 16671 raft_consensus.cc:348] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:22.838300 16671 raft_consensus.cc:729] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c874074d234b159322f057cbc4d217, State: Initialized, Role: FOLLOWER
I20240502 08:05:22.838836 16671 consensus_queue.cc:260] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:22.839004 16671 raft_consensus.cc:388] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:22.839133 16671 raft_consensus.cc:482] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:22.839241 16671 raft_consensus.cc:3037] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:22.841468 16671 raft_consensus.cc:504] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:22.841851 16671 leader_election.cc:302] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62c874074d234b159322f057cbc4d217; no voters:
I20240502 08:05:22.842196 16671 leader_election.cc:288] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 08:05:22.842314 16784 raft_consensus.cc:2781] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:22.842512 16784 raft_consensus.cc:686] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [term 2 LEADER]: Becoming Leader. State: Replica: 62c874074d234b159322f057cbc4d217, State: Running, Role: LEADER
I20240502 08:05:22.842904 16779 heartbeater.cc:503] Master 127.16.36.254:46305 was elected leader, sending a full tablet report...
I20240502 08:05:22.842821 16784 consensus_queue.cc:237] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } }
I20240502 08:05:22.843067 16671 ts_tablet_manager.cc:1398] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20240502 08:05:22.845306 15922 catalog_manager.cc:5484] T c4a48d1332154a1ba61019567823d00e P 62c874074d234b159322f057cbc4d217 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "62c874074d234b159322f057cbc4d217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62c874074d234b159322f057cbc4d217" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 36037 } } }
W20240502 08:05:22.940301 16782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:22.940586 16782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:22.940640 16782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:22.943879 16782 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20240502 08:05:22.943960 16782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:22.944056 16782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 08:05:22.947997 16782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:33465
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=36905
--tserver_master_addrs=127.16.36.254:46305
--never_fsync=true
--heap_profile_path=/tmp/kudu.16782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:22.949025 16782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:22.950187 16782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:22.959038 16793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.959465 16794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:22.960064 16796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:22.960956 16782 server_base.cc:1034] running on GCE node
I20240502 08:05:22.961369 16782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:22.962070 16782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:22.963282 16782 hybrid_clock.cc:648] HybridClock initialized: now 1714637122963266 us; error 38 us; skew 500 ppm
I20240502 08:05:22.965775 16782 webserver.cc:460] Webserver started at http://127.16.36.196:36905/ using document root <none> and password file <none>
I20240502 08:05:22.966322 16782 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:22.966389 16782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:22.969931 16782 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:22.972143 16802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:22.973217 16782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:22.973335 16782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dce9d531641c45abb70201c9fa6b66ca"
format_stamp: "Formatted at 2024-05-02 08:05:17 on dist-test-slave-9sjq"
I20240502 08:05:22.973845 16782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:23.003366 16782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:23.004055 16782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:23.004207 16782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:23.004822 16782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:23.005800 16782 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:23.005859 16782 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:23.005936 16782 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:23.005960 16782 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:23.014396 16782 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:33465
I20240502 08:05:23.014442 16915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:33465 every 8 connection(s)
I20240502 08:05:23.015451 16782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:05:23.022862 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 16782
I20240502 08:05:23.025163 16916 heartbeater.cc:347] Connected to a master server at 127.16.36.254:46305
I20240502 08:05:23.025436 16916 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:23.025913 16916 heartbeater.cc:511] Master 127.16.36.254:46305 requested a full tablet report, sending...
I20240502 08:05:23.026867 15922 ts_manager.cc:194] Re-registered known tserver with Master: dce9d531641c45abb70201c9fa6b66ca (127.16.36.196:33465)
I20240502 08:05:23.027416 15922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:34717
I20240502 08:05:23.221820 16045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.223304 16177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.225086 16850 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:23.228727 16713 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:23.322450 16177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.323210 16850 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:23.323760 16045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.324175 16713 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:23.438158 16713 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:23.439057 16177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.439540 16850 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:23.451821 16045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.541115 16713 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:23.541384 16045 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 08:05:23.541646 16850 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:23.542191 16177 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ec543b338b9f47fcb97c6c0f1cfeae9e | 127.16.36.254:46305 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.212:33405 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
4669623dcbe9447ab43d30b1c7713228 | 127.16.36.194:38567 | HEALTHY | <none> | 6 | 0
62c874074d234b159322f057cbc4d217 | 127.16.36.195:36037 | HEALTHY | <none> | 1 | 0
dce9d531641c45abb70201c9fa6b66ca | 127.16.36.196:33465 | HEALTHY | <none> | 0 | 0
e92df7bd75144c989f39ece64a410221 | 127.16.36.193:41969 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.16.36.193 | experimental | 127.16.36.193:41969
local_ip_for_outbound_sockets | 127.16.36.194 | experimental | 127.16.36.194:38567
local_ip_for_outbound_sockets | 127.16.36.195 | experimental | 127.16.36.195:36037
local_ip_for_outbound_sockets | 127.16.36.196 | experimental | 127.16.36.196:33465
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | false | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.16.36.193:41969
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.16.36.194:38567
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.16.36.195:36037
server_dump_info_path | /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_0.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.16.36.196:33465
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.16.36.212:33405 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 6
Third Quartile | 6
Maximum | 6
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 08:05:23.567551 16531 log_verifier.cc:126] Checking tablet 0cba4b7f334f4687a32d43b2442d5609
I20240502 08:05:23.575152 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 0cba4b7f334f4687a32d43b2442d5609
I20240502 08:05:23.575215 16531 log_verifier.cc:126] Checking tablet 3a226ca918c64c169624cf6dfb283b23
I20240502 08:05:23.582716 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3a226ca918c64c169624cf6dfb283b23
I20240502 08:05:23.582782 16531 log_verifier.cc:126] Checking tablet 3e0f7c4d3b8f432399ebd5bf9e81a59a
I20240502 08:05:23.589882 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 3e0f7c4d3b8f432399ebd5bf9e81a59a
I20240502 08:05:23.589946 16531 log_verifier.cc:126] Checking tablet 98f0148fa9be40e58ecfb4b330975a6d
I20240502 08:05:23.597486 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 98f0148fa9be40e58ecfb4b330975a6d
I20240502 08:05:23.597556 16531 log_verifier.cc:126] Checking tablet a30076a542cc4fa88d97991a9bd107ec
I20240502 08:05:23.605238 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a30076a542cc4fa88d97991a9bd107ec
I20240502 08:05:23.605304 16531 log_verifier.cc:126] Checking tablet a7b5fa8de1fd4977bf31eb7231da4cea
I20240502 08:05:23.612705 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet a7b5fa8de1fd4977bf31eb7231da4cea
I20240502 08:05:23.612771 16531 log_verifier.cc:126] Checking tablet c0a02f15e22842a1b46060531feb0312
I20240502 08:05:23.619969 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c0a02f15e22842a1b46060531feb0312
I20240502 08:05:23.620038 16531 log_verifier.cc:126] Checking tablet c3afa575296949f7b139ffe2a2f53a9c
I20240502 08:05:23.627470 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c3afa575296949f7b139ffe2a2f53a9c
I20240502 08:05:23.627543 16531 log_verifier.cc:126] Checking tablet c4a48d1332154a1ba61019567823d00e
I20240502 08:05:23.634728 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c4a48d1332154a1ba61019567823d00e
I20240502 08:05:23.634805 16531 log_verifier.cc:126] Checking tablet d5145f37eceb4bca96b2d0d7bd1e842e
I20240502 08:05:23.642084 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet d5145f37eceb4bca96b2d0d7bd1e842e
I20240502 08:05:23.642153 16531 log_verifier.cc:126] Checking tablet dadd09184ab54e1bb3caf0ce781553a1
I20240502 08:05:23.649508 16531 log_verifier.cc:177] Verified matching terms for 2 ops in tablet dadd09184ab54e1bb3caf0ce781553a1
I20240502 08:05:23.649580 16531 log_verifier.cc:126] Checking tablet e66666a128de46239c8bd33b13ef22ff
I20240502 08:05:23.656940 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet e66666a128de46239c8bd33b13ef22ff
I20240502 08:05:23.657006 16531 log_verifier.cc:126] Checking tablet f2ba3bf250534204839874d30fdb3d08
I20240502 08:05:23.664129 16531 log_verifier.cc:177] Verified matching terms for 1 ops in tablet f2ba3bf250534204839874d30fdb3d08
I20240502 08:05:23.664875 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 15983
I20240502 08:05:23.671259 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16114
I20240502 08:05:23.677356 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16650
I20240502 08:05:23.683001 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 16782
I20240502 08:05:23.688762 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 15892
2024-05-02T08:05:23Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/0 (6720 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (6720 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2
I20240502 08:05:23.722970 16531 test_util.cc:268] Using random seed: 279813844
I20240502 08:05:23.723773 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:05:23.723812 16531 ts_itest-base.cc:113] --------------
I20240502 08:05:23.723824 16531 ts_itest-base.cc:114] 5 tablet servers
I20240502 08:05:23.723834 16531 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:05:23.723845 16531 ts_itest-base.cc:116] --------------
2024-05-02T08:05:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:05:23Z Disabled control of system clock
I20240502 08:05:23.732762 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:43721
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:43721
--location_mapping_cmd=/tmp/dist-test-taskm5AjVx/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 08:05:23.840406 17062 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:05:23.840696 17062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:23.840723 17062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:23.840761 17062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:23.843724 17062 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:23.843801 17062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:05:23.843827 17062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:23.843845 17062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:05:23.843863 17062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:05:23.847559 17062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskm5AjVx/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:2
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:43721
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:43721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.17062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:23.848448 17062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:23.849553 17062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:23.857082 17070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:23.857578 17068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:23.858299 17067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:23.859138 17062 server_base.cc:1034] running on GCE node
I20240502 08:05:23.859616 17062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:23.860559 17062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:23.861729 17062 hybrid_clock.cc:648] HybridClock initialized: now 1714637123861707 us; error 35 us; skew 500 ppm
I20240502 08:05:23.863656 17062 webserver.cc:460] Webserver started at http://127.16.36.254:36869/ using document root <none> and password file <none>
I20240502 08:05:23.864180 17062 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:23.864235 17062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:23.864418 17062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:23.866106 17062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4e3c3cf820fc4942a7ddaf89ef181be2"
format_stamp: "Formatted at 2024-05-02 08:05:23 on dist-test-slave-9sjq"
I20240502 08:05:23.866545 17062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4e3c3cf820fc4942a7ddaf89ef181be2"
format_stamp: "Formatted at 2024-05-02 08:05:23 on dist-test-slave-9sjq"
I20240502 08:05:23.869848 17062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:05:23.872026 17076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:23.873116 17062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:23.873226 17062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4e3c3cf820fc4942a7ddaf89ef181be2"
format_stamp: "Formatted at 2024-05-02 08:05:23 on dist-test-slave-9sjq"
I20240502 08:05:23.873318 17062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:23.890496 17062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:23.891156 17062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:23.891295 17062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:23.897723 17062 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:43721
I20240502 08:05:23.897733 17128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:43721 every 8 connection(s)
I20240502 08:05:23.898728 17062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:05:23.901290 17129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:23.906402 17129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2: Bootstrap starting.
I20240502 08:05:23.908239 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17062
I20240502 08:05:23.908375 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:05:23.908738 17129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:23.909413 17129 log.cc:824] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:23.911253 17129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2: No bootstrap required, opened a new log
I20240502 08:05:23.914391 17129 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 43721 } }
I20240502 08:05:23.914628 17129 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:23.914718 17129 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3c3cf820fc4942a7ddaf89ef181be2, State: Initialized, Role: FOLLOWER
I20240502 08:05:23.915148 17129 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 43721 } }
I20240502 08:05:23.915274 17129 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:23.915335 17129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:23.915432 17129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:23.916265 17129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 43721 } }
I20240502 08:05:23.916630 17129 leader_election.cc:302] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e3c3cf820fc4942a7ddaf89ef181be2; no voters:
I20240502 08:05:23.916960 17129 leader_election.cc:288] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:23.917371 17134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:23.917572 17134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3c3cf820fc4942a7ddaf89ef181be2, State: Running, Role: LEADER
I20240502 08:05:23.917863 17134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 43721 } }
I20240502 08:05:23.918118 17129 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:05:23.918924 17136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e3c3cf820fc4942a7ddaf89ef181be2. Latest consensus state: current_term: 1 leader_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 43721 } } }
I20240502 08:05:23.919072 17136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:23.919292 17135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e3c3cf820fc4942a7ddaf89ef181be2" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 43721 } } }
I20240502 08:05:23.919391 17135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2 [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:23.919492 17143 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:05:23.922516 17143 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:05:23.927956 17143 catalog_manager.cc:1347] Generated new cluster ID: d138d27ada454559aa408f5ddcf81597
I20240502 08:05:23.928030 17143 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:05:23.957430 17143 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:05:23.958434 17143 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:05:23.968724 17143 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4e3c3cf820fc4942a7ddaf89ef181be2: Generated new TSK 0
I20240502 08:05:23.969516 17143 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:05:23.981969 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:05:24.096203 17153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:24.096478 17153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:24.096542 17153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:24.099654 17153 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:24.099731 17153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:24.099812 17153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 08:05:24.103772 17153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.17153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:24.104825 17153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:24.105988 17153 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:24.114750 17158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.115265 17159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.115680 17161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:24.116575 17153 server_base.cc:1034] running on GCE node
I20240502 08:05:24.117002 17153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:24.117746 17153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:24.118903 17153 hybrid_clock.cc:648] HybridClock initialized: now 1714637124118884 us; error 65 us; skew 500 ppm
I20240502 08:05:24.120859 17153 webserver.cc:460] Webserver started at http://127.16.36.193:37451/ using document root <none> and password file <none>
I20240502 08:05:24.121485 17153 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:24.121554 17153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:24.121733 17153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:24.123473 17153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.123924 17153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.127367 17153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20240502 08:05:24.129758 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.130824 17153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:24.130939 17153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.131060 17153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:24.150947 17153 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:24.151625 17153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:24.151784 17153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:24.152361 17153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:24.153316 17153 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:24.153367 17153 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.153430 17153 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:24.153456 17153 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.162007 17153 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:46425
I20240502 08:05:24.162020 17280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:46425 every 8 connection(s)
I20240502 08:05:24.162955 17153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:05:24.168545 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17153
I20240502 08:05:24.168656 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:05:24.170998 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:24.172207 17281 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:24.172497 17281 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:24.173053 17281 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:24.213922 17093 ts_manager.cc:194] Registered new tserver with Master: d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:24.215291 17093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:51005
W20240502 08:05:24.283676 17284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:24.283931 17284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:24.284004 17284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:24.287089 17284 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:24.287175 17284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:24.287278 17284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 08:05:24.291002 17284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.17284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:24.291983 17284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:24.293196 17284 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:24.301616 17290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.302078 17291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.302670 17293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:24.303453 17284 server_base.cc:1034] running on GCE node
I20240502 08:05:24.303843 17284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:24.304515 17284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:24.305671 17284 hybrid_clock.cc:648] HybridClock initialized: now 1714637124305659 us; error 29 us; skew 500 ppm
I20240502 08:05:24.307819 17284 webserver.cc:460] Webserver started at http://127.16.36.194:44495/ using document root <none> and password file <none>
I20240502 08:05:24.308382 17284 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:24.308446 17284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:24.308679 17284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:24.310406 17284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d8da072ac9514d3097eb441c8ee68428"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.310890 17284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d8da072ac9514d3097eb441c8ee68428"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.314299 17284 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 08:05:24.316473 17299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.317456 17284 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:24.317566 17284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d8da072ac9514d3097eb441c8ee68428"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.317656 17284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:24.331223 17284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:24.331851 17284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:24.332012 17284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:24.332577 17284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:24.333469 17284 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:24.333518 17284 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.333557 17284 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:24.333591 17284 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.341706 17284 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:35347
I20240502 08:05:24.341745 17412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:35347 every 8 connection(s)
I20240502 08:05:24.342648 17284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:05:24.347126 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17284
I20240502 08:05:24.347275 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:05:24.349793 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:24.352823 17413 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:24.353053 17413 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:24.353636 17413 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:24.387836 17093 ts_manager.cc:194] Registered new tserver with Master: d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:24.388819 17093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:40007
W20240502 08:05:24.460673 17416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:24.460907 17416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:24.460953 17416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:24.464000 17416 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:24.464075 17416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:24.464145 17416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 08:05:24.467737 17416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.17416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:24.468698 17416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:24.469882 17416 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:24.478240 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.478799 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.479355 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:24.480242 17416 server_base.cc:1034] running on GCE node
I20240502 08:05:24.480731 17416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:24.481398 17416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:24.482569 17416 hybrid_clock.cc:648] HybridClock initialized: now 1714637124482554 us; error 32 us; skew 500 ppm
I20240502 08:05:24.484403 17416 webserver.cc:460] Webserver started at http://127.16.36.195:33647/ using document root <none> and password file <none>
I20240502 08:05:24.484988 17416 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:24.485061 17416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:24.485240 17416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:24.486922 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1256e2e5163d4240bd7dd791512ffa5c"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.487416 17416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1256e2e5163d4240bd7dd791512ffa5c"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.490787 17416 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240502 08:05:24.493074 17431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.494105 17416 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:24.494220 17416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1256e2e5163d4240bd7dd791512ffa5c"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.494309 17416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:24.507599 17416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:24.508248 17416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:24.508428 17416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:24.509011 17416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:24.509918 17416 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:24.509970 17416 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.510040 17416 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:24.510066 17416 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.518579 17416 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:41337
I20240502 08:05:24.518617 17544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:41337 every 8 connection(s)
I20240502 08:05:24.519599 17416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:05:24.526443 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17416
I20240502 08:05:24.526573 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:05:24.528708 17545 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:24.528941 17545 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:24.529016 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:24.529502 17545 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:24.577601 17093 ts_manager.cc:194] Registered new tserver with Master: 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:24.578330 17093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:60313
W20240502 08:05:24.652928 17548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:24.653177 17548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:24.653256 17548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:24.656301 17548 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:24.656383 17548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:24.656531 17548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 08:05:24.660116 17548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.17548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:24.661087 17548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:24.662236 17548 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:24.671015 17555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.671562 17557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.672124 17554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:24.673027 17548 server_base.cc:1034] running on GCE node
I20240502 08:05:24.673446 17548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:24.674119 17548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:24.675289 17548 hybrid_clock.cc:648] HybridClock initialized: now 1714637124675265 us; error 37 us; skew 500 ppm
I20240502 08:05:24.677193 17548 webserver.cc:460] Webserver started at http://127.16.36.196:45979/ using document root <none> and password file <none>
I20240502 08:05:24.677760 17548 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:24.677817 17548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:24.677968 17548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:24.679569 17548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7e6d75f768764435ab5f0b63159ca43a"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.680007 17548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7e6d75f768764435ab5f0b63159ca43a"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.683283 17548 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240502 08:05:24.685492 17563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.686512 17548 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:24.686623 17548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7e6d75f768764435ab5f0b63159ca43a"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.686715 17548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:24.705893 17548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:24.706526 17548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:24.706696 17548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:24.707226 17548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:24.708101 17548 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:24.708151 17548 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.708210 17548 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:24.708259 17548 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.716816 17548 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:38989
I20240502 08:05:24.716837 17676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:38989 every 8 connection(s)
I20240502 08:05:24.717756 17548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:05:24.725265 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17548
I20240502 08:05:24.725400 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:05:24.727141 17677 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:24.727358 17677 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:24.727655 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:0
--local_ip_for_outbound_sockets=127.16.36.197
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:24.727905 17677 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:24.767971 17093 ts_manager.cc:194] Registered new tserver with Master: 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989)
I20240502 08:05:24.769001 17093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:39347
W20240502 08:05:24.841984 17680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:24.842216 17680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:24.842260 17680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:24.845263 17680 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:24.845340 17680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:24.845408 17680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 08:05:24.849016 17680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=0
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.17680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:24.849918 17680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:24.851014 17680 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:24.859592 17686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.859943 17687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:24.860706 17689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:24.861454 17680 server_base.cc:1034] running on GCE node
I20240502 08:05:24.861871 17680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:24.862538 17680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:24.863700 17680 hybrid_clock.cc:648] HybridClock initialized: now 1714637124863682 us; error 38 us; skew 500 ppm
I20240502 08:05:24.865662 17680 webserver.cc:460] Webserver started at http://127.16.36.197:39459/ using document root <none> and password file <none>
I20240502 08:05:24.866222 17680 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:24.866278 17680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:24.866434 17680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:24.868110 17680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "93ae1f0181ba4eae9801831b0e037f6d"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.868633 17680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "93ae1f0181ba4eae9801831b0e037f6d"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.871939 17680 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:24.874433 17695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.875499 17680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 08:05:24.875612 17680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "93ae1f0181ba4eae9801831b0e037f6d"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:24.875720 17680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:24.888324 17680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:24.888981 17680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:24.889149 17680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:24.889657 17680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:24.890542 17680 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:24.890589 17680 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.890635 17680 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:24.890664 17680 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:24.899469 17680 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:44867
I20240502 08:05:24.899516 17808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:44867 every 8 connection(s)
I20240502 08:05:24.900378 17680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:05:24.903564 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 17680
I20240502 08:05:24.903705 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 08:05:24.909854 17809 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:24.910076 17809 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:24.910598 17809 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:24.945925 17093 ts_manager.cc:194] Registered new tserver with Master: 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867)
I20240502 08:05:24.946595 17093 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:33139
I20240502 08:05:24.954540 16531 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:05:24.964349 16531 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 08:05:24.964434 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17680
I20240502 08:05:24.971061 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17548
I20240502 08:05:25.218385 17281 heartbeater.cc:503] Master 127.16.36.254:43721 was elected leader, sending a full tablet report...
I20240502 08:05:25.391360 17413 heartbeater.cc:503] Master 127.16.36.254:43721 was elected leader, sending a full tablet report...
I20240502 08:05:25.580906 17545 heartbeater.cc:503] Master 127.16.36.254:43721 was elected leader, sending a full tablet report...
I20240502 08:05:28.728623 17093 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34192:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 08:05:28.732081 17093 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:28.748534 17345 tablet_service.cc:1467] Processing CreateTablet for tablet c095f330e9d941108ceb48b65c4e4327 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.749644 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c095f330e9d941108ceb48b65c4e4327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.750416 17477 tablet_service.cc:1467] Processing CreateTablet for tablet c095f330e9d941108ceb48b65c4e4327 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.750416 17478 tablet_service.cc:1467] Processing CreateTablet for tablet 4735728a9541425c8482ed7fb17bd483 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.751317 17479 tablet_service.cc:1467] Processing CreateTablet for tablet df7c076e0edb4c9089c58c4a3d55eb83 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.751381 17478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4735728a9541425c8482ed7fb17bd483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.748534 17346 tablet_service.cc:1467] Processing CreateTablet for tablet 4735728a9541425c8482ed7fb17bd483 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.751806 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4735728a9541425c8482ed7fb17bd483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.751847 17215 tablet_service.cc:1467] Processing CreateTablet for tablet df7c076e0edb4c9089c58c4a3d55eb83 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.752745 17214 tablet_service.cc:1467] Processing CreateTablet for tablet 4735728a9541425c8482ed7fb17bd483 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.753163 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4735728a9541425c8482ed7fb17bd483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.753336 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7c076e0edb4c9089c58c4a3d55eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.748534 17347 tablet_service.cc:1467] Processing CreateTablet for tablet df7c076e0edb4c9089c58c4a3d55eb83 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.755117 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7c076e0edb4c9089c58c4a3d55eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.757257 17828 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:28.757688 17829 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:28.758198 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c095f330e9d941108ceb48b65c4e4327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.759436 17213 tablet_service.cc:1467] Processing CreateTablet for tablet c095f330e9d941108ceb48b65c4e4327 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:28.759536 17828 tablet_bootstrap.cc:654] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.759717 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c095f330e9d941108ceb48b65c4e4327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.760685 17829 tablet_bootstrap.cc:654] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.760814 17828 log.cc:824] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:28.762573 17828 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:28.762724 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7c076e0edb4c9089c58c4a3d55eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:28.762750 17828 ts_tablet_manager.cc:1367] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 08:05:28.762959 17829 log.cc:824] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:28.764847 17830 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:28.765043 17829 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:28.765324 17829 ts_tablet_manager.cc:1367] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 08:05:28.765709 17828 raft_consensus.cc:348] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.765918 17828 raft_consensus.cc:374] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.765983 17828 raft_consensus.cc:729] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.766412 17828 consensus_queue.cc:260] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.767313 17828 ts_tablet_manager.cc:1398] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.004s user 0.005s sys 0.001s
I20240502 08:05:28.767619 17828 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:28.766913 17830 tablet_bootstrap.cc:654] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.768568 17828 tablet_bootstrap.cc:654] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.768700 17830 log.cc:824] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:28.769140 17829 raft_consensus.cc:348] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.769435 17829 raft_consensus.cc:374] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.769506 17829 raft_consensus.cc:729] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.769645 17828 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:28.769714 17828 ts_tablet_manager.cc:1367] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:05:28.770220 17828 raft_consensus.cc:348] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.770315 17828 raft_consensus.cc:374] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.770345 17830 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:28.770349 17828 raft_consensus.cc:729] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.770296 17829 consensus_queue.cc:260] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.770462 17828 consensus_queue.cc:260] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.770511 17830 ts_tablet_manager.cc:1367] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 08:05:28.770730 17828 ts_tablet_manager.cc:1398] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:28.770872 17828 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:28.772097 17829 ts_tablet_manager.cc:1398] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240502 08:05:28.772254 17828 tablet_bootstrap.cc:654] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.772491 17829 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:28.773412 17828 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:28.773499 17828 ts_tablet_manager.cc:1367] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:05:28.773953 17828 raft_consensus.cc:348] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.774075 17828 raft_consensus.cc:374] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.774107 17828 raft_consensus.cc:729] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
W20240502 08:05:28.774196 17546 tablet.cc:2369] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:05:28.774230 17828 consensus_queue.cc:260] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.774461 17828 ts_tablet_manager.cc:1398] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:28.774858 17829 tablet_bootstrap.cc:654] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.775463 17830 raft_consensus.cc:348] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.775686 17830 raft_consensus.cc:374] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.775753 17830 raft_consensus.cc:729] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.776214 17830 consensus_queue.cc:260] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.776612 17829 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:28.776701 17829 ts_tablet_manager.cc:1367] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:05:28.777130 17830 ts_tablet_manager.cc:1398] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20240502 08:05:28.777276 17829 raft_consensus.cc:348] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.777365 17830 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:28.777377 17829 raft_consensus.cc:374] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.777412 17829 raft_consensus.cc:729] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.777565 17829 consensus_queue.cc:260] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.777868 17829 ts_tablet_manager.cc:1398] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:28.778039 17829 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:28.778307 17830 tablet_bootstrap.cc:654] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.778805 17829 tablet_bootstrap.cc:654] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.779507 17830 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:28.779593 17830 ts_tablet_manager.cc:1367] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:05:28.780140 17830 raft_consensus.cc:348] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.780285 17830 raft_consensus.cc:374] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.780332 17830 raft_consensus.cc:729] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.780493 17830 consensus_queue.cc:260] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.780774 17830 ts_tablet_manager.cc:1398] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:28.780946 17830 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:28.781143 17829 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:28.781213 17829 ts_tablet_manager.cc:1367] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:05:28.781612 17829 raft_consensus.cc:348] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.781709 17829 raft_consensus.cc:374] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.781745 17829 raft_consensus.cc:729] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.781793 17830 tablet_bootstrap.cc:654] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:28.781874 17829 consensus_queue.cc:260] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.782135 17829 ts_tablet_manager.cc:1398] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:28.782877 17830 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:28.782963 17830 ts_tablet_manager.cc:1367] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:28.783404 17830 raft_consensus.cc:348] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.783527 17830 raft_consensus.cc:374] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:28.783560 17830 raft_consensus.cc:729] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:28.783718 17830 consensus_queue.cc:260] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.783986 17830 ts_tablet_manager.cc:1398] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:28.821890 17839 raft_consensus.cc:482] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:28.822105 17839 raft_consensus.cc:504] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.823372 17839 leader_election.cc:288] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:28.827754 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c095f330e9d941108ceb48b65c4e4327" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428" is_pre_election: true
I20240502 08:05:28.827929 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c095f330e9d941108ceb48b65c4e4327" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c" is_pre_election: true
I20240502 08:05:28.828044 17367 raft_consensus.cc:2454] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 0.
I20240502 08:05:28.828199 17499 raft_consensus.cc:2454] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 0.
I20240502 08:05:28.828524 17169 leader_election.cc:302] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67b9894d6dc4495ad18c22ee4a0d458, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:28.828754 17839 raft_consensus.cc:2781] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:28.828830 17839 raft_consensus.cc:482] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:28.828859 17839 raft_consensus.cc:3037] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:28.829763 17839 raft_consensus.cc:504] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.830117 17839 leader_election.cc:288] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 election: Requested vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:28.830523 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c095f330e9d941108ceb48b65c4e4327" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:28.830581 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c095f330e9d941108ceb48b65c4e4327" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:28.830650 17367 raft_consensus.cc:3037] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:28.830731 17499 raft_consensus.cc:3037] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:28.831643 17499 raft_consensus.cc:2454] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 1.
I20240502 08:05:28.831961 17169 leader_election.cc:302] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d67b9894d6dc4495ad18c22ee4a0d458; no voters:
I20240502 08:05:28.832024 17367 raft_consensus.cc:2454] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 1.
I20240502 08:05:28.832149 17839 raft_consensus.cc:2781] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:28.832404 17839 raft_consensus.cc:686] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Becoming Leader. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Running, Role: LEADER
I20240502 08:05:28.832978 17839 consensus_queue.cc:237] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:28.835880 17092 catalog_manager.cc:5484] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: term changed from 0 to 1, leader changed from <none> to d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:28.847142 17414 tablet.cc:2369] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:05:28.917778 17282 tablet.cc:2369] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:05:29.026906 17836 raft_consensus.cc:482] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:29.027058 17836 raft_consensus.cc:504] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.028172 17836 leader_election.cc:288] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:29.032346 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4735728a9541425c8482ed7fb17bd483" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428" is_pre_election: true
I20240502 08:05:29.032356 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4735728a9541425c8482ed7fb17bd483" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:29.032531 17367 raft_consensus.cc:2454] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.032594 17235 raft_consensus.cc:2454] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.032888 17433 leader_election.cc:302] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.033185 17836 raft_consensus.cc:2781] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:29.033252 17836 raft_consensus.cc:482] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:29.033291 17836 raft_consensus.cc:3037] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.034178 17836 raft_consensus.cc:504] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.034525 17836 leader_election.cc:288] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Requested vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:29.034830 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4735728a9541425c8482ed7fb17bd483" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:29.034912 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4735728a9541425c8482ed7fb17bd483" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:29.034972 17235 raft_consensus.cc:3037] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.035055 17367 raft_consensus.cc:3037] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.035822 17367 raft_consensus.cc:2454] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.035874 17235 raft_consensus.cc:2454] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.036114 17433 leader_election.cc:302] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.036302 17836 raft_consensus.cc:2781] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:29.036562 17836 raft_consensus.cc:686] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Becoming Leader. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Running, Role: LEADER
I20240502 08:05:29.036914 17836 consensus_queue.cc:237] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.039772 17092 catalog_manager.cc:5484] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195). New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:29.123075 17836 raft_consensus.cc:482] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:29.123231 17836 raft_consensus.cc:504] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.123679 17836 leader_election.cc:288] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:29.124140 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428" is_pre_election: true
I20240502 08:05:29.124171 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:29.124293 17367 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.124295 17235 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.124609 17433 leader_election.cc:302] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.124773 17836 raft_consensus.cc:2781] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:29.124830 17836 raft_consensus.cc:482] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:29.124879 17836 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.125675 17836 raft_consensus.cc:504] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.125977 17836 leader_election.cc:288] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Requested vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:29.126382 17235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:29.126392 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:29.126500 17367 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.126499 17235 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.127254 17367 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.127288 17235 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.127537 17433 leader_election.cc:302] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d67b9894d6dc4495ad18c22ee4a0d458; no voters:
I20240502 08:05:29.127714 17836 raft_consensus.cc:2781] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:29.127799 17836 raft_consensus.cc:686] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Becoming Leader. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Running, Role: LEADER
I20240502 08:05:29.127956 17836 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.129678 17092 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195). New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:29.270273 17839 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.274637 17839 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 08:05:29.318743 17852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:29.319039 17852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:29.322367 17852 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 08:05:29.415043 17367 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.415234 17235 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.415242 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.415427 17366 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.416131 17849 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.416535 17849 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.416790 17849 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.417013 17849 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.428578 17855 mvcc.cc:205] Tried to move back new op lower bound from 7023153682078908416 to 7023153680909520896. Current Snapshot: MvccSnapshot[applied={T|T < 7023153682078908416}]
I20240502 08:05:29.437635 17891 mvcc.cc:205] Tried to move back new op lower bound from 7023153682078908416 to 7023153680909520896. Current Snapshot: MvccSnapshot[applied={T|T < 7023153682078908416}]
I20240502 08:05:29.437671 17854 mvcc.cc:205] Tried to move back new op lower bound from 7023153682078908416 to 7023153680909520896. Current Snapshot: MvccSnapshot[applied={T|T < 7023153682078908416}]
INSERT report
rows total: 2000
time total: 109.309 ms
time per row: 0.0546544 ms
I20240502 08:05:29.481112 17090 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34192:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 08:05:29.481599 17090 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:29.486682 17347 tablet_service.cc:1467] Processing CreateTablet for tablet efbde964902249719b24cf743249ec6f (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.486992 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbde964902249719b24cf743249ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.486959 17477 tablet_service.cc:1467] Processing CreateTablet for tablet efbde964902249719b24cf743249ec6f (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.487205 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbde964902249719b24cf743249ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.487305 17215 tablet_service.cc:1467] Processing CreateTablet for tablet efbde964902249719b24cf743249ec6f (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.487566 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbde964902249719b24cf743249ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.486997 17346 tablet_service.cc:1467] Processing CreateTablet for tablet 84a7fcaa3ee34406827fcbabfda1aed5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.488759 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a7fcaa3ee34406827fcbabfda1aed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.489609 17912 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:29.490053 17214 tablet_service.cc:1467] Processing CreateTablet for tablet a76ee285ba8645a88251bd351c4e4146 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.490255 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76ee285ba8645a88251bd351c4e4146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.490496 17912 tablet_bootstrap.cc:654] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.490483 17347 tablet_service.cc:1467] Processing CreateTablet for tablet a76ee285ba8645a88251bd351c4e4146 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.490684 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76ee285ba8645a88251bd351c4e4146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.491168 17479 tablet_service.cc:1467] Processing CreateTablet for tablet a76ee285ba8645a88251bd351c4e4146 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.491410 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76ee285ba8645a88251bd351c4e4146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.492249 17215 tablet_service.cc:1467] Processing CreateTablet for tablet 84a7fcaa3ee34406827fcbabfda1aed5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.492576 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a7fcaa3ee34406827fcbabfda1aed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.492882 17910 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:29.493068 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 84a7fcaa3ee34406827fcbabfda1aed5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:29.493294 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a7fcaa3ee34406827fcbabfda1aed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:29.493712 17910 tablet_bootstrap.cc:654] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.494601 17911 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:29.495357 17911 tablet_bootstrap.cc:654] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.495468 17912 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:29.495568 17912 ts_tablet_manager.cc:1367] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240502 08:05:29.495703 17910 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:29.495793 17910 ts_tablet_manager.cc:1367] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:29.496029 17912 raft_consensus.cc:348] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.496131 17912 raft_consensus.cc:374] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.496167 17912 raft_consensus.cc:729] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.496290 17910 raft_consensus.cc:348] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.496330 17912 consensus_queue.cc:260] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.496387 17910 raft_consensus.cc:374] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.496421 17910 raft_consensus.cc:729] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.496521 17911 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:29.496592 17911 ts_tablet_manager.cc:1367] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:29.496562 17910 consensus_queue.cc:260] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.496654 17912 ts_tablet_manager.cc:1398] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.496829 17912 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:29.496928 17910 ts_tablet_manager.cc:1398] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:29.497005 17911 raft_consensus.cc:348] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.497087 17910 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:29.497097 17911 raft_consensus.cc:374] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.497128 17911 raft_consensus.cc:729] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.497280 17911 consensus_queue.cc:260] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.497612 17911 ts_tablet_manager.cc:1398] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.497737 17912 tablet_bootstrap.cc:654] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.497784 17911 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:29.497949 17910 tablet_bootstrap.cc:654] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.498821 17911 tablet_bootstrap.cc:654] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.499867 17911 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:29.499867 17912 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:29.499951 17911 ts_tablet_manager.cc:1367] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:29.499951 17912 ts_tablet_manager.cc:1367] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:29.500349 17912 raft_consensus.cc:348] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.500349 17911 raft_consensus.cc:348] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.500479 17911 raft_consensus.cc:374] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.500478 17912 raft_consensus.cc:374] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.500515 17912 raft_consensus.cc:729] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.500515 17911 raft_consensus.cc:729] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.500526 17910 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:29.500597 17910 ts_tablet_manager.cc:1367] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240502 08:05:29.500641 17911 consensus_queue.cc:260] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.500656 17912 consensus_queue.cc:260] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.500892 17911 ts_tablet_manager.cc:1398] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.500972 17912 ts_tablet_manager.cc:1398] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.501036 17911 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:29.501000 17910 raft_consensus.cc:348] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.501108 17912 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:29.501101 17910 raft_consensus.cc:374] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.501139 17910 raft_consensus.cc:729] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.501251 17910 consensus_queue.cc:260] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.501539 17910 ts_tablet_manager.cc:1398] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.501703 17910 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:29.502003 17912 tablet_bootstrap.cc:654] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.502254 17911 tablet_bootstrap.cc:654] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.502615 17910 tablet_bootstrap.cc:654] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:29.503706 17910 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:29.503794 17910 ts_tablet_manager.cc:1367] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:05:29.504112 17911 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:29.504171 17911 ts_tablet_manager.cc:1367] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:29.504215 17910 raft_consensus.cc:348] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.504307 17910 raft_consensus.cc:374] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.504338 17910 raft_consensus.cc:729] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.504402 17912 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:29.504489 17910 consensus_queue.cc:260] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.504540 17912 ts_tablet_manager.cc:1367] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:29.504523 17911 raft_consensus.cc:348] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.504596 17911 raft_consensus.cc:374] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.504626 17911 raft_consensus.cc:729] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.504755 17910 ts_tablet_manager.cc:1398] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.504747 17911 consensus_queue.cc:260] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.504994 17911 ts_tablet_manager.cc:1398] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.504981 17912 raft_consensus.cc:348] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.505079 17912 raft_consensus.cc:374] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:29.505111 17912 raft_consensus.cc:729] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:29.505239 17912 consensus_queue.cc:260] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.506027 17912 ts_tablet_manager.cc:1398] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:29.507104 17887 raft_consensus.cc:482] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:29.507205 17887 raft_consensus.cc:504] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.507575 17887 leader_election.cc:288] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:29.507997 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbde964902249719b24cf743249ec6f" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:29.508075 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbde964902249719b24cf743249ec6f" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428" is_pre_election: true
I20240502 08:05:29.508117 17234 raft_consensus.cc:2454] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.508193 17367 raft_consensus.cc:2454] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.508400 17433 leader_election.cc:302] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d67b9894d6dc4495ad18c22ee4a0d458; no voters:
I20240502 08:05:29.508702 17887 raft_consensus.cc:2781] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:29.508761 17887 raft_consensus.cc:482] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:29.508808 17887 raft_consensus.cc:3037] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.509690 17887 raft_consensus.cc:504] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.509984 17887 leader_election.cc:288] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Requested vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:29.510367 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbde964902249719b24cf743249ec6f" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:29.510450 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efbde964902249719b24cf743249ec6f" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:29.510490 17234 raft_consensus.cc:3037] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.510557 17367 raft_consensus.cc:3037] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.511514 17367 raft_consensus.cc:2454] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.511514 17234 raft_consensus.cc:2454] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.511807 17433 leader_election.cc:302] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.511973 17887 raft_consensus.cc:2781] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:29.512068 17887 raft_consensus.cc:686] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Becoming Leader. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Running, Role: LEADER
I20240502 08:05:29.512224 17887 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.513845 17084 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195). New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:29.603370 17856 raft_consensus.cc:482] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:29.603514 17856 raft_consensus.cc:504] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.604552 17856 leader_election.cc:288] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:29.608112 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76ee285ba8645a88251bd351c4e4146" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:29.608264 17234 raft_consensus.cc:2454] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 0.
I20240502 08:05:29.608364 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76ee285ba8645a88251bd351c4e4146" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c" is_pre_election: true
I20240502 08:05:29.608551 17499 raft_consensus.cc:2454] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 0.
I20240502 08:05:29.608649 17301 leader_election.cc:302] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67b9894d6dc4495ad18c22ee4a0d458, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.608904 17856 raft_consensus.cc:2781] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:29.608971 17856 raft_consensus.cc:482] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:29.609000 17856 raft_consensus.cc:3037] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.609833 17856 raft_consensus.cc:504] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.610163 17856 leader_election.cc:288] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 election: Requested vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:29.610553 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76ee285ba8645a88251bd351c4e4146" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:29.610534 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a76ee285ba8645a88251bd351c4e4146" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:29.610652 17499 raft_consensus.cc:3037] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.610670 17234 raft_consensus.cc:3037] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.611387 17499 raft_consensus.cc:2454] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 1.
I20240502 08:05:29.611471 17234 raft_consensus.cc:2454] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 1.
I20240502 08:05:29.611670 17301 leader_election.cc:302] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.611836 17856 raft_consensus.cc:2781] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:29.612059 17856 raft_consensus.cc:686] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Becoming Leader. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Running, Role: LEADER
I20240502 08:05:29.612337 17856 consensus_queue.cc:237] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.614677 17084 catalog_manager.cc:5484] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8da072ac9514d3097eb441c8ee68428 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:29.734109 17887 raft_consensus.cc:482] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:29.734247 17887 raft_consensus.cc:504] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.734709 17887 leader_election.cc:288] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:29.735148 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a7fcaa3ee34406827fcbabfda1aed5" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428" is_pre_election: true
I20240502 08:05:29.735173 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a7fcaa3ee34406827fcbabfda1aed5" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:29.735322 17234 raft_consensus.cc:2454] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.735358 17367 raft_consensus.cc:2454] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 0.
I20240502 08:05:29.735667 17433 leader_election.cc:302] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.735869 17887 raft_consensus.cc:2781] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:29.735944 17887 raft_consensus.cc:482] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:29.735996 17887 raft_consensus.cc:3037] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.736887 17887 raft_consensus.cc:504] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.737221 17887 leader_election.cc:288] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Requested vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:29.737609 17367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a7fcaa3ee34406827fcbabfda1aed5" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:29.737663 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84a7fcaa3ee34406827fcbabfda1aed5" candidate_uuid: "1256e2e5163d4240bd7dd791512ffa5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:29.737751 17367 raft_consensus.cc:3037] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.737782 17234 raft_consensus.cc:3037] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:29.738726 17367 raft_consensus.cc:2454] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.738781 17234 raft_consensus.cc:2454] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1256e2e5163d4240bd7dd791512ffa5c in term 1.
I20240502 08:05:29.739022 17433 leader_election.cc:302] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:29.739194 17887 raft_consensus.cc:2781] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:29.739293 17887 raft_consensus.cc:686] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Becoming Leader. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Running, Role: LEADER
I20240502 08:05:29.739436 17887 consensus_queue.cc:237] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:29.741633 17084 catalog_manager.cc:5484] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195). New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } }
W20240502 08:05:29.898679 17928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:29.898941 17928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:29.901978 17928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:05:29.974938 17887 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.979956 17234 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.979930 17836 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.980350 17367 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.982033 17887 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.982327 17887 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.990235 17499 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.990815 17925 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:29.996826 17234 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:29.997334 17925 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 99.7476 ms
time per row: 0.0498738 ms
I20240502 08:05:30.046902 17091 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34192:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 08:05:30.047484 17091 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:30.052506 17215 tablet_service.cc:1467] Processing CreateTablet for tablet 38d81695cf9d4c1faf2b67580d44480e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.052512 17347 tablet_service.cc:1467] Processing CreateTablet for tablet 38d81695cf9d4c1faf2b67580d44480e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.052735 17477 tablet_service.cc:1467] Processing CreateTablet for tablet 38d81695cf9d4c1faf2b67580d44480e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.052822 17215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d81695cf9d4c1faf2b67580d44480e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.052822 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d81695cf9d4c1faf2b67580d44480e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.053046 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d81695cf9d4c1faf2b67580d44480e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.054095 17213 tablet_service.cc:1467] Processing CreateTablet for tablet 4da8a9287c5b48ac9e990ecaac40fc9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.054333 17213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da8a9287c5b48ac9e990ecaac40fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.054582 17346 tablet_service.cc:1467] Processing CreateTablet for tablet c0180a09549b4556addb20ce1f8d4e63 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.054827 17346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0180a09549b4556addb20ce1f8d4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.054582 17345 tablet_service.cc:1467] Processing CreateTablet for tablet 4da8a9287c5b48ac9e990ecaac40fc9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.055899 17214 tablet_service.cc:1467] Processing CreateTablet for tablet c0180a09549b4556addb20ce1f8d4e63 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.056051 17345 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da8a9287c5b48ac9e990ecaac40fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.056107 17214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0180a09549b4556addb20ce1f8d4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.057562 17986 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:30.057667 17479 tablet_service.cc:1467] Processing CreateTablet for tablet 4da8a9287c5b48ac9e990ecaac40fc9e (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.057922 17479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da8a9287c5b48ac9e990ecaac40fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.058207 17477 tablet_service.cc:1467] Processing CreateTablet for tablet c0180a09549b4556addb20ce1f8d4e63 (DEFAULT_TABLE table=rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:30.058421 17477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0180a09549b4556addb20ce1f8d4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:30.058512 17985 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:30.058558 17986 tablet_bootstrap.cc:654] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.059480 17985 tablet_bootstrap.cc:654] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.059597 17986 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:30.059671 17986 ts_tablet_manager.cc:1367] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:30.060125 17986 raft_consensus.cc:348] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.060245 17986 raft_consensus.cc:374] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.060287 17986 raft_consensus.cc:729] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.060436 17986 consensus_queue.cc:260] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.060770 17985 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:30.060777 17986 ts_tablet_manager.cc:1398] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:30.060846 17985 ts_tablet_manager.cc:1367] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:30.060938 17986 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:30.061309 17985 raft_consensus.cc:348] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.061408 17985 raft_consensus.cc:374] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.061439 17985 raft_consensus.cc:729] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.061579 17985 consensus_queue.cc:260] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.061775 17986 tablet_bootstrap.cc:654] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.061861 17987 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:30.062817 17987 tablet_bootstrap.cc:654] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.062841 17985 ts_tablet_manager.cc:1398] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:05:30.063042 17985 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:30.063819 17985 tablet_bootstrap.cc:654] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.064409 17986 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:30.064518 17986 ts_tablet_manager.cc:1367] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:05:30.064616 17987 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:30.064690 17987 ts_tablet_manager.cc:1367] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:30.064893 17985 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:30.064946 17985 ts_tablet_manager.cc:1367] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:05:30.064958 17986 raft_consensus.cc:348] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.065052 17986 raft_consensus.cc:374] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.065085 17986 raft_consensus.cc:729] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.065096 17987 raft_consensus.cc:348] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.065187 17987 raft_consensus.cc:374] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.065220 17987 raft_consensus.cc:729] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.065207 17986 consensus_queue.cc:260] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.065340 17987 consensus_queue.cc:260] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.065583 17987 ts_tablet_manager.cc:1398] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:30.065605 17985 raft_consensus.cc:348] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.065732 17985 raft_consensus.cc:374] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.065753 17987 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:30.065765 17985 raft_consensus.cc:729] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.065869 17985 consensus_queue.cc:260] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.066140 17985 ts_tablet_manager.cc:1398] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:30.066314 17985 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:30.066632 17987 tablet_bootstrap.cc:654] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.067157 17985 tablet_bootstrap.cc:654] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.067389 17986 ts_tablet_manager.cc:1398] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:05:30.067574 17986 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:30.068122 17985 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: No bootstrap required, opened a new log
I20240502 08:05:30.068122 17987 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:30.068197 17985 ts_tablet_manager.cc:1367] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:05:30.068197 17987 ts_tablet_manager.cc:1367] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:30.068493 17986 tablet_bootstrap.cc:654] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.068609 17987 raft_consensus.cc:348] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.068609 17985 raft_consensus.cc:348] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.068701 17987 raft_consensus.cc:374] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.068712 17985 raft_consensus.cc:374] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.068745 17985 raft_consensus.cc:729] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.068737 17987 raft_consensus.cc:729] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.068859 17985 consensus_queue.cc:260] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.068883 17987 consensus_queue.cc:260] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.069121 17985 ts_tablet_manager.cc:1398] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:30.069180 17987 ts_tablet_manager.cc:1398] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:30.069418 17987 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:30.070255 17986 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c: No bootstrap required, opened a new log
I20240502 08:05:30.070355 17986 ts_tablet_manager.cc:1367] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:30.070863 17986 raft_consensus.cc:348] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.071043 17986 raft_consensus.cc:374] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.071112 17987 tablet_bootstrap.cc:654] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:30.071134 17986 raft_consensus.cc:729] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.071350 17986 consensus_queue.cc:260] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.071683 17986 ts_tablet_manager.cc:1398] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:30.072288 17987 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: No bootstrap required, opened a new log
I20240502 08:05:30.072382 17987 ts_tablet_manager.cc:1367] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:30.072827 17987 raft_consensus.cc:348] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.072917 17987 raft_consensus.cc:374] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:30.072950 17987 raft_consensus.cc:729] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: FOLLOWER
I20240502 08:05:30.073074 17987 consensus_queue.cc:260] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.073329 17987 ts_tablet_manager.cc:1398] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:30.169615 17856 raft_consensus.cc:482] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:30.169780 17856 raft_consensus.cc:504] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.170254 17856 leader_election.cc:288] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:30.170730 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:30.170790 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c" is_pre_election: true
I20240502 08:05:30.170871 17234 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 0.
I20240502 08:05:30.170941 17499 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 0.
I20240502 08:05:30.171219 17301 leader_election.cc:302] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67b9894d6dc4495ad18c22ee4a0d458, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:30.171396 17856 raft_consensus.cc:2781] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:30.171458 17856 raft_consensus.cc:482] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:30.171490 17856 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.172629 17856 raft_consensus.cc:504] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.173013 17856 leader_election.cc:288] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 election: Requested vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:30.173419 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:30.173494 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:30.173566 17234 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.173615 17499 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.174619 17499 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 1.
I20240502 08:05:30.174619 17234 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 1.
I20240502 08:05:30.174978 17301 leader_election.cc:302] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:30.175158 17856 raft_consensus.cc:2781] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:30.175292 17856 raft_consensus.cc:686] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Becoming Leader. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Running, Role: LEADER
I20240502 08:05:30.175432 17856 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.177258 17084 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8da072ac9514d3097eb441c8ee68428 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:30.183560 17856 raft_consensus.cc:482] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:30.183709 17856 raft_consensus.cc:504] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.184136 17856 leader_election.cc:288] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:30.184784 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da8a9287c5b48ac9e990ecaac40fc9e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" is_pre_election: true
I20240502 08:05:30.184791 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da8a9287c5b48ac9e990ecaac40fc9e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c" is_pre_election: true
I20240502 08:05:30.184958 17234 raft_consensus.cc:2454] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 0.
I20240502 08:05:30.184967 17499 raft_consensus.cc:2454] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 0.
I20240502 08:05:30.185484 17301 leader_election.cc:302] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67b9894d6dc4495ad18c22ee4a0d458, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:30.185686 17856 raft_consensus.cc:2781] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:30.185788 17856 raft_consensus.cc:482] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:30.185843 17856 raft_consensus.cc:3037] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.186894 17856 raft_consensus.cc:504] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.187255 17856 leader_election.cc:288] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 election: Requested vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:30.187647 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da8a9287c5b48ac9e990ecaac40fc9e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:30.187716 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da8a9287c5b48ac9e990ecaac40fc9e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:30.187772 17234 raft_consensus.cc:3037] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.187849 17499 raft_consensus.cc:3037] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.188628 17234 raft_consensus.cc:2454] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 1.
I20240502 08:05:30.188793 17499 raft_consensus.cc:2454] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 1.
I20240502 08:05:30.188947 17301 leader_election.cc:302] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67b9894d6dc4495ad18c22ee4a0d458, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:30.189127 17856 raft_consensus.cc:2781] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:30.189250 17856 raft_consensus.cc:686] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Becoming Leader. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Running, Role: LEADER
I20240502 08:05:30.189406 17856 consensus_queue.cc:237] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.191263 17084 catalog_manager.cc:5484] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8da072ac9514d3097eb441c8ee68428 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:30.377070 17845 raft_consensus.cc:482] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:30.377218 17845 raft_consensus.cc:504] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.377703 17845 leader_election.cc:288] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:30.378302 17365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428" is_pre_election: true
I20240502 08:05:30.378289 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c" is_pre_election: true
I20240502 08:05:30.378455 17499 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 0.
I20240502 08:05:30.378455 17365 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 0.
I20240502 08:05:30.378762 17169 leader_election.cc:302] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, d67b9894d6dc4495ad18c22ee4a0d458; no voters:
I20240502 08:05:30.378971 17845 raft_consensus.cc:2781] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:30.379030 17845 raft_consensus.cc:482] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:30.379081 17845 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.379914 17845 raft_consensus.cc:504] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.380234 17845 leader_election.cc:288] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 election: Requested vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:30.380623 17365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:30.380699 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:30.380759 17365 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.380864 17499 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:30.381767 17365 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 1.
I20240502 08:05:30.381767 17499 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d67b9894d6dc4495ad18c22ee4a0d458 in term 1.
I20240502 08:05:30.382081 17169 leader_election.cc:302] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d67b9894d6dc4495ad18c22ee4a0d458, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:30.382239 17845 raft_consensus.cc:2781] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:30.382417 17845 raft_consensus.cc:686] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Becoming Leader. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Running, Role: LEADER
I20240502 08:05:30.382565 17845 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } }
I20240502 08:05:30.384231 17084 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: term changed from 0 to 1, leader changed from <none> to d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193). New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:30.544986 18001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:30.545259 18001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:30.548578 18001 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:05:30.586560 17856 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:30.601922 17856 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:30.635514 17234 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:30.635825 17499 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:30.635941 17498 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:30.636433 17367 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:30.636466 17925 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:30.637089 17925 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:30.637686 17999 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:30.637993 17999 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 114.345 ms
time per row: 0.0571727 ms
I20240502 08:05:30.706372 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.197:44867
--local_ip_for_outbound_sockets=127.16.36.197
--tserver_master_addrs=127.16.36.254:43721
--webserver_port=39459
--webserver_interface=127.16.36.197
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:05:30.815308 18057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:30.815608 18057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:30.815681 18057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:30.819129 18057 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:30.819202 18057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:30.819303 18057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.197
I20240502 08:05:30.823230 18057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.197:44867
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.16.36.197
--webserver_port=39459
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.18057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.197
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:30.824229 18057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:30.825470 18057 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240502 08:05:30.834754 18057 server_base.cc:1034] running on GCE node
W20240502 08:05:30.835218 18065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:30.835777 18062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:30.836244 18063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:30.836717 18057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:30.837397 18057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:30.838570 18057 hybrid_clock.cc:648] HybridClock initialized: now 1714637130838563 us; error 26 us; skew 500 ppm
I20240502 08:05:30.840340 18057 webserver.cc:460] Webserver started at http://127.16.36.197:39459/ using document root <none> and password file <none>
I20240502 08:05:30.840921 18057 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:30.840999 18057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:30.844329 18057 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:30.846180 18071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:30.847218 18057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:30.847339 18057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "93ae1f0181ba4eae9801831b0e037f6d"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:30.847743 18057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:30.860208 18057 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:30.860865 18057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:30.861006 18057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:30.861477 18057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:30.862299 18057 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:30.862350 18057 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:30.862390 18057 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:30.862411 18057 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:30.870607 18057 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.197:44867
I20240502 08:05:30.870671 18184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.197:44867 every 8 connection(s)
I20240502 08:05:30.871619 18057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 08:05:30.872186 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 18057
I20240502 08:05:30.872705 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:38989
--local_ip_for_outbound_sockets=127.16.36.196
--tserver_master_addrs=127.16.36.254:43721
--webserver_port=45979
--webserver_interface=127.16.36.196
--builtin_ntp_servers=127.16.36.212:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:30.881242 18185 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:30.881474 18185 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:30.882155 18185 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:30.883314 17092 ts_manager.cc:194] Re-registered known tserver with Master: 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867)
I20240502 08:05:30.883971 17092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.197:60753
W20240502 08:05:30.985250 18187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:30.985472 18187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:30.985515 18187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:30.988396 18187 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:30.988471 18187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:30.988576 18187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 08:05:30.993026 18187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:38989
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=45979
--tserver_master_addrs=127.16.36.254:43721
--never_fsync=true
--heap_profile_path=/tmp/kudu.18187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:30.993888 18187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:30.994959 18187 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:31.003481 18194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:31.004033 18196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:31.004573 18193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:31.005364 18187 server_base.cc:1034] running on GCE node
I20240502 08:05:31.005722 18187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:31.006390 18187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:31.007561 18187 hybrid_clock.cc:648] HybridClock initialized: now 1714637131007541 us; error 40 us; skew 500 ppm
I20240502 08:05:31.009426 18187 webserver.cc:460] Webserver started at http://127.16.36.196:45979/ using document root <none> and password file <none>
I20240502 08:05:31.009987 18187 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:31.010054 18187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:31.013370 18187 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:31.015331 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:31.016345 18187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:31.016438 18187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7e6d75f768764435ab5f0b63159ca43a"
format_stamp: "Formatted at 2024-05-02 08:05:24 on dist-test-slave-9sjq"
I20240502 08:05:31.016893 18187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:31.035005 18187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:31.035707 18187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:31.035856 18187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:31.036355 18187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:31.037374 18187 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:31.037432 18187 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:31.037482 18187 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:31.037509 18187 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:31.046667 18187 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:38989
I20240502 08:05:31.046689 18315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:38989 every 8 connection(s)
I20240502 08:05:31.047744 18187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:05:31.048841 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 18187
I20240502 08:05:31.058110 18316 heartbeater.cc:347] Connected to a master server at 127.16.36.254:43721
I20240502 08:05:31.058403 18316 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:31.059110 18316 heartbeater.cc:511] Master 127.16.36.254:43721 requested a full tablet report, sending...
I20240502 08:05:31.060247 17092 ts_manager.cc:194] Re-registered known tserver with Master: 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989)
I20240502 08:05:31.060814 17092 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:47253
I20240502 08:05:31.252406 17477 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:31.252846 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.256919 18119 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.257638 17214 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:31.264959 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:31.355201 17477 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:31.355237 17214 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:31.357435 18119 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.358273 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.368804 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:31.471390 17477 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:31.472028 18119 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.472901 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.473228 17214 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:31.474040 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:31.570916 17477 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:31.571295 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.571324 18119 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:31.572512 17214 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:31.576663 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:31.617718 17234 consensus_queue.cc:237] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:31.619899 17367 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.619925 17499 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.620378 18464 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.620627 18047 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.623148 18463 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.623617 17499 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
W20240502 08:05:31.624434 17169 consensus_peers.cc:485] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 -> Peer 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989): Couldn't send request to peer 7e6d75f768764435ab5f0b63159ca43a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c095f330e9d941108ceb48b65c4e4327. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.624341 17367 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.626081 17091 catalog_manager.cc:5484] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:31.635679 17367 consensus_queue.cc:237] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:31.637560 17499 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.637616 17234 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.638007 18478 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.638247 18054 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.640192 18478 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.640558 17499 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.640684 17234 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
W20240502 08:05:31.642797 17301 consensus_peers.cc:485] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 -> Peer 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867): Couldn't send request to peer 93ae1f0181ba4eae9801831b0e037f6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4da8a9287c5b48ac9e990ecaac40fc9e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.643656 17092 catalog_manager.cc:5484] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:31.652191 17499 consensus_queue.cc:237] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:31.654254 17367 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.654316 17234 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.654711 18486 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.654955 17973 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.657099 18487 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.657583 17234 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.657889 17367 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
W20240502 08:05:31.658660 17433 consensus_peers.cc:485] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c -> Peer 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989): Couldn't send request to peer 7e6d75f768764435ab5f0b63159ca43a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84a7fcaa3ee34406827fcbabfda1aed5. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.660167 17091 catalog_manager.cc:5484] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:31.667599 17367 consensus_queue.cc:237] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:31.669071 17234 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:05:31.669131 17301 consensus_peers.cc:485] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 -> Peer 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867): Couldn't send request to peer 93ae1f0181ba4eae9801831b0e037f6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a76ee285ba8645a88251bd351c4e4146. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.669466 18478 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.669673 17499 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.670029 18478 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.671248 18477 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.671633 17234 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.674381 17084 catalog_manager.cc:5484] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.674937 17499 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.681998 17367 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:31.683579 17234 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.683614 17499 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.684010 18478 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.684244 18499 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.685921 18054 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.686506 17499 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
W20240502 08:05:31.686940 17301 consensus_peers.cc:485] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 -> Peer 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989): Couldn't send request to peer 7e6d75f768764435ab5f0b63159ca43a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c0180a09549b4556addb20ce1f8d4e63. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.688750 17092 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:31.688827 17234 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.697001 17499 consensus_queue.cc:237] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:31.698745 17367 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.699029 17234 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.699254 18487 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.699602 18487 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.701114 17973 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
W20240502 08:05:31.702761 17433 consensus_peers.cc:485] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c -> Peer 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867): Couldn't send request to peer 93ae1f0181ba4eae9801831b0e037f6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4735728a9541425c8482ed7fb17bd483. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.702872 17367 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.703253 17234 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:31.703550 17091 catalog_manager.cc:5484] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:31.712867 17499 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:31.714315 17234 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:31.714315 17367 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:05:31.714555 17433 consensus_peers.cc:485] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c -> Peer 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989): Couldn't send request to peer 7e6d75f768764435ab5f0b63159ca43a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: efbde964902249719b24cf743249ec6f. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:31.714785 18487 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.714982 18486 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:31.716969 17973 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.717432 17367 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.717458 17234 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:31.720144 17092 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:31.886324 18185 heartbeater.cc:503] Master 127.16.36.254:43721 was elected leader, sending a full tablet report...
I20240502 08:05:32.063233 18316 heartbeater.cc:503] Master 127.16.36.254:43721 was elected leader, sending a full tablet report...
I20240502 08:05:32.070683 18592 ts_tablet_manager.cc:902] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Initiating tablet copy from peer d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:32.071427 18592 tablet_copy_client.cc:308] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:35347
I20240502 08:05:32.075304 17387 tablet_copy_service.cc:140] P d8da072ac9514d3097eb441c8ee68428: Received BeginTabletCopySession request for tablet a76ee285ba8645a88251bd351c4e4146 from peer 93ae1f0181ba4eae9801831b0e037f6d ({username='slave'} at 127.16.36.197:45489)
I20240502 08:05:32.075474 17387 tablet_copy_service.cc:161] P d8da072ac9514d3097eb441c8ee68428: Beginning new tablet copy session on tablet a76ee285ba8645a88251bd351c4e4146 from peer 93ae1f0181ba4eae9801831b0e037f6d at {username='slave'} at 127.16.36.197:45489: session id = 93ae1f0181ba4eae9801831b0e037f6d-a76ee285ba8645a88251bd351c4e4146
I20240502 08:05:32.076838 17387 tablet_copy_source_session.cc:205] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.080374 18592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a76ee285ba8645a88251bd351c4e4146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.081105 18598 ts_tablet_manager.cc:902] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: Initiating tablet copy from peer d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:32.081784 18598 tablet_copy_client.cc:308] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:35347
I20240502 08:05:32.082283 17387 tablet_copy_service.cc:140] P d8da072ac9514d3097eb441c8ee68428: Received BeginTabletCopySession request for tablet 4da8a9287c5b48ac9e990ecaac40fc9e from peer 93ae1f0181ba4eae9801831b0e037f6d ({username='slave'} at 127.16.36.197:45489)
I20240502 08:05:32.082391 17387 tablet_copy_service.cc:161] P d8da072ac9514d3097eb441c8ee68428: Beginning new tablet copy session on tablet 4da8a9287c5b48ac9e990ecaac40fc9e from peer 93ae1f0181ba4eae9801831b0e037f6d at {username='slave'} at 127.16.36.197:45489: session id = 93ae1f0181ba4eae9801831b0e037f6d-4da8a9287c5b48ac9e990ecaac40fc9e
I20240502 08:05:32.083314 17387 tablet_copy_source_session.cc:205] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.084177 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da8a9287c5b48ac9e990ecaac40fc9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.086367 18598 tablet_copy_client.cc:783] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.086390 18592 tablet_copy_client.cc:783] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.086556 18598 tablet_copy_client.cc:647] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.086603 18592 tablet_copy_client.cc:647] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.088366 18598 tablet_copy_client.cc:523] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.088366 18592 tablet_copy_client.cc:523] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.090157 18598 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap starting.
I20240502 08:05:32.090157 18592 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap starting.
I20240502 08:05:32.090312 18603 ts_tablet_manager.cc:902] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Initiating tablet copy from peer 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:32.091133 18603 tablet_copy_client.cc:308] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:41337
I20240502 08:05:32.095875 18592 log.cc:824] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:32.096886 17519 tablet_copy_service.cc:140] P 1256e2e5163d4240bd7dd791512ffa5c: Received BeginTabletCopySession request for tablet efbde964902249719b24cf743249ec6f from peer 7e6d75f768764435ab5f0b63159ca43a ({username='slave'} at 127.16.36.196:41729)
I20240502 08:05:32.097029 17519 tablet_copy_service.cc:161] P 1256e2e5163d4240bd7dd791512ffa5c: Beginning new tablet copy session on tablet efbde964902249719b24cf743249ec6f from peer 7e6d75f768764435ab5f0b63159ca43a at {username='slave'} at 127.16.36.196:41729: session id = 7e6d75f768764435ab5f0b63159ca43a-efbde964902249719b24cf743249ec6f
I20240502 08:05:32.098209 17519 tablet_copy_source_session.cc:205] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.101383 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbde964902249719b24cf743249ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.106717 18603 tablet_copy_client.cc:783] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.106967 18603 tablet_copy_client.cc:647] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.108750 18603 tablet_copy_client.cc:523] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.110651 18603 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap starting.
I20240502 08:05:32.116562 18603 log.cc:824] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:32.125676 18592 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.125676 18598 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.126096 18592 tablet_bootstrap.cc:492] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap complete.
I20240502 08:05:32.126096 18598 tablet_bootstrap.cc:492] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap complete.
I20240502 08:05:32.126468 18592 ts_tablet_manager.cc:1367] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Time spent bootstrapping tablet: real 0.036s user 0.033s sys 0.000s
I20240502 08:05:32.126468 18598 ts_tablet_manager.cc:1367] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20240502 08:05:32.128943 18598 raft_consensus.cc:348] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:32.128943 18592 raft_consensus.cc:348] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:32.129283 18598 raft_consensus.cc:729] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93ae1f0181ba4eae9801831b0e037f6d, State: Initialized, Role: LEARNER
I20240502 08:05:32.129288 18592 raft_consensus.cc:729] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93ae1f0181ba4eae9801831b0e037f6d, State: Initialized, Role: LEARNER
I20240502 08:05:32.129730 18598 consensus_queue.cc:260] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:32.129730 18592 consensus_queue.cc:260] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:32.130671 18592 ts_tablet_manager.cc:1398] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:05:32.130671 18598 ts_tablet_manager.cc:1398] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:05:32.131645 17386 tablet_copy_service.cc:342] P d8da072ac9514d3097eb441c8ee68428: Request end of tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-a76ee285ba8645a88251bd351c4e4146 received from {username='slave'} at 127.16.36.197:45489
I20240502 08:05:32.131758 17386 tablet_copy_service.cc:433] P d8da072ac9514d3097eb441c8ee68428: ending tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-a76ee285ba8645a88251bd351c4e4146 on tablet a76ee285ba8645a88251bd351c4e4146 with peer 93ae1f0181ba4eae9801831b0e037f6d
I20240502 08:05:32.132084 17387 tablet_copy_service.cc:342] P d8da072ac9514d3097eb441c8ee68428: Request end of tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-4da8a9287c5b48ac9e990ecaac40fc9e received from {username='slave'} at 127.16.36.197:45489
I20240502 08:05:32.132158 17387 tablet_copy_service.cc:433] P d8da072ac9514d3097eb441c8ee68428: ending tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-4da8a9287c5b48ac9e990ecaac40fc9e on tablet 4da8a9287c5b48ac9e990ecaac40fc9e with peer 93ae1f0181ba4eae9801831b0e037f6d
I20240502 08:05:32.140436 18603 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.140861 18603 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap complete.
I20240502 08:05:32.141177 18603 ts_tablet_manager.cc:1367] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.001s
I20240502 08:05:32.143234 18603 raft_consensus.cc:348] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.143489 18603 raft_consensus.cc:729] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e6d75f768764435ab5f0b63159ca43a, State: Initialized, Role: LEARNER
I20240502 08:05:32.143831 18603 consensus_queue.cc:260] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.144743 18603 ts_tablet_manager.cc:1398] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 08:05:32.145661 17519 tablet_copy_service.cc:342] P 1256e2e5163d4240bd7dd791512ffa5c: Request end of tablet copy session 7e6d75f768764435ab5f0b63159ca43a-efbde964902249719b24cf743249ec6f received from {username='slave'} at 127.16.36.196:41729
I20240502 08:05:32.145778 17519 tablet_copy_service.cc:433] P 1256e2e5163d4240bd7dd791512ffa5c: ending tablet copy session 7e6d75f768764435ab5f0b63159ca43a-efbde964902249719b24cf743249ec6f on tablet efbde964902249719b24cf743249ec6f with peer 7e6d75f768764435ab5f0b63159ca43a
I20240502 08:05:32.174011 18603 ts_tablet_manager.cc:902] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: Initiating tablet copy from peer d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:32.174535 18603 tablet_copy_client.cc:308] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46425
I20240502 08:05:32.178668 17255 tablet_copy_service.cc:140] P d67b9894d6dc4495ad18c22ee4a0d458: Received BeginTabletCopySession request for tablet c095f330e9d941108ceb48b65c4e4327 from peer 7e6d75f768764435ab5f0b63159ca43a ({username='slave'} at 127.16.36.196:46541)
I20240502 08:05:32.178864 17255 tablet_copy_service.cc:161] P d67b9894d6dc4495ad18c22ee4a0d458: Beginning new tablet copy session on tablet c095f330e9d941108ceb48b65c4e4327 from peer 7e6d75f768764435ab5f0b63159ca43a at {username='slave'} at 127.16.36.196:46541: session id = 7e6d75f768764435ab5f0b63159ca43a-c095f330e9d941108ceb48b65c4e4327
I20240502 08:05:32.180100 17255 tablet_copy_source_session.cc:205] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.181416 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c095f330e9d941108ceb48b65c4e4327. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.183280 18603 tablet_copy_client.cc:783] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.183436 18603 tablet_copy_client.cc:647] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.184895 18603 tablet_copy_client.cc:523] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.185902 18603 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap starting.
I20240502 08:05:32.189654 18598 ts_tablet_manager.cc:902] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: Initiating tablet copy from peer 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:32.190104 18598 tablet_copy_client.cc:308] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:41337
I20240502 08:05:32.193624 17519 tablet_copy_service.cc:140] P 1256e2e5163d4240bd7dd791512ffa5c: Received BeginTabletCopySession request for tablet 4735728a9541425c8482ed7fb17bd483 from peer 93ae1f0181ba4eae9801831b0e037f6d ({username='slave'} at 127.16.36.197:33181)
I20240502 08:05:32.193770 17519 tablet_copy_service.cc:161] P 1256e2e5163d4240bd7dd791512ffa5c: Beginning new tablet copy session on tablet 4735728a9541425c8482ed7fb17bd483 from peer 93ae1f0181ba4eae9801831b0e037f6d at {username='slave'} at 127.16.36.197:33181: session id = 93ae1f0181ba4eae9801831b0e037f6d-4735728a9541425c8482ed7fb17bd483
I20240502 08:05:32.193848 18624 ts_tablet_manager.cc:902] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: Initiating tablet copy from peer 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:32.194401 18624 tablet_copy_client.cc:308] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:41337
I20240502 08:05:32.194687 17519 tablet_copy_source_session.cc:205] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.194906 17518 tablet_copy_service.cc:140] P 1256e2e5163d4240bd7dd791512ffa5c: Received BeginTabletCopySession request for tablet 84a7fcaa3ee34406827fcbabfda1aed5 from peer 7e6d75f768764435ab5f0b63159ca43a ({username='slave'} at 127.16.36.196:41729)
I20240502 08:05:32.194993 17518 tablet_copy_service.cc:161] P 1256e2e5163d4240bd7dd791512ffa5c: Beginning new tablet copy session on tablet 84a7fcaa3ee34406827fcbabfda1aed5 from peer 7e6d75f768764435ab5f0b63159ca43a at {username='slave'} at 127.16.36.196:41729: session id = 7e6d75f768764435ab5f0b63159ca43a-84a7fcaa3ee34406827fcbabfda1aed5
I20240502 08:05:32.195596 18598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4735728a9541425c8482ed7fb17bd483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.195880 17518 tablet_copy_source_session.cc:205] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.196797 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84a7fcaa3ee34406827fcbabfda1aed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.197980 18598 tablet_copy_client.cc:783] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.198124 18598 tablet_copy_client.cc:647] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.198980 18624 tablet_copy_client.cc:783] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.199121 18624 tablet_copy_client.cc:647] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.199565 18598 tablet_copy_client.cc:523] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.200574 18598 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap starting.
I20240502 08:05:32.201799 18624 tablet_copy_client.cc:523] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.202988 18624 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap starting.
I20240502 08:05:32.207268 18603 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.207537 18603 tablet_bootstrap.cc:492] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap complete.
I20240502 08:05:32.207752 18603 ts_tablet_manager.cc:1367] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: Time spent bootstrapping tablet: real 0.022s user 0.022s sys 0.000s
I20240502 08:05:32.208391 18603 raft_consensus.cc:348] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.208546 18603 raft_consensus.cc:729] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e6d75f768764435ab5f0b63159ca43a, State: Initialized, Role: LEARNER
I20240502 08:05:32.208690 18603 consensus_queue.cc:260] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.209000 18603 ts_tablet_manager.cc:1398] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:32.210207 17255 tablet_copy_service.cc:342] P d67b9894d6dc4495ad18c22ee4a0d458: Request end of tablet copy session 7e6d75f768764435ab5f0b63159ca43a-c095f330e9d941108ceb48b65c4e4327 received from {username='slave'} at 127.16.36.196:46541
I20240502 08:05:32.210325 17255 tablet_copy_service.cc:433] P d67b9894d6dc4495ad18c22ee4a0d458: ending tablet copy session 7e6d75f768764435ab5f0b63159ca43a-c095f330e9d941108ceb48b65c4e4327 on tablet c095f330e9d941108ceb48b65c4e4327 with peer 7e6d75f768764435ab5f0b63159ca43a
I20240502 08:05:32.219388 18603 ts_tablet_manager.cc:902] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: Initiating tablet copy from peer d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:32.219830 18603 tablet_copy_client.cc:308] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:35347
I20240502 08:05:32.224989 18598 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.225004 17387 tablet_copy_service.cc:140] P d8da072ac9514d3097eb441c8ee68428: Received BeginTabletCopySession request for tablet c0180a09549b4556addb20ce1f8d4e63 from peer 7e6d75f768764435ab5f0b63159ca43a ({username='slave'} at 127.16.36.196:37419)
I20240502 08:05:32.225253 17387 tablet_copy_service.cc:161] P d8da072ac9514d3097eb441c8ee68428: Beginning new tablet copy session on tablet c0180a09549b4556addb20ce1f8d4e63 from peer 7e6d75f768764435ab5f0b63159ca43a at {username='slave'} at 127.16.36.196:37419: session id = 7e6d75f768764435ab5f0b63159ca43a-c0180a09549b4556addb20ce1f8d4e63
I20240502 08:05:32.225323 18598 tablet_bootstrap.cc:492] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap complete.
I20240502 08:05:32.225490 18598 ts_tablet_manager.cc:1367] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: Time spent bootstrapping tablet: real 0.025s user 0.023s sys 0.000s
I20240502 08:05:32.225996 18598 raft_consensus.cc:348] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:32.226120 18598 raft_consensus.cc:729] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93ae1f0181ba4eae9801831b0e037f6d, State: Initialized, Role: LEARNER
I20240502 08:05:32.226215 18598 consensus_queue.cc:260] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:32.226279 17387 tablet_copy_source_session.cc:205] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:32.226492 18598 ts_tablet_manager.cc:1398] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:32.226975 17518 tablet_copy_service.cc:342] P 1256e2e5163d4240bd7dd791512ffa5c: Request end of tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-4735728a9541425c8482ed7fb17bd483 received from {username='slave'} at 127.16.36.197:33181
I20240502 08:05:32.227057 17518 tablet_copy_service.cc:433] P 1256e2e5163d4240bd7dd791512ffa5c: ending tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-4735728a9541425c8482ed7fb17bd483 on tablet 4735728a9541425c8482ed7fb17bd483 with peer 93ae1f0181ba4eae9801831b0e037f6d
I20240502 08:05:32.227614 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0180a09549b4556addb20ce1f8d4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:32.229564 18603 tablet_copy_client.cc:783] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:32.229725 18603 tablet_copy_client.cc:647] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:32.230865 18603 tablet_copy_client.cc:523] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:32.232115 18603 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap starting.
I20240502 08:05:32.238009 18624 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.238278 18624 tablet_bootstrap.cc:492] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap complete.
I20240502 08:05:32.238500 18624 ts_tablet_manager.cc:1367] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.000s
I20240502 08:05:32.238986 18624 raft_consensus.cc:348] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.239156 18624 raft_consensus.cc:729] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e6d75f768764435ab5f0b63159ca43a, State: Initialized, Role: LEARNER
I20240502 08:05:32.239269 18624 consensus_queue.cc:260] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.239647 18624 ts_tablet_manager.cc:1398] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:32.240183 17518 tablet_copy_service.cc:342] P 1256e2e5163d4240bd7dd791512ffa5c: Request end of tablet copy session 7e6d75f768764435ab5f0b63159ca43a-84a7fcaa3ee34406827fcbabfda1aed5 received from {username='slave'} at 127.16.36.196:41729
I20240502 08:05:32.240281 17518 tablet_copy_service.cc:433] P 1256e2e5163d4240bd7dd791512ffa5c: ending tablet copy session 7e6d75f768764435ab5f0b63159ca43a-84a7fcaa3ee34406827fcbabfda1aed5 on tablet 84a7fcaa3ee34406827fcbabfda1aed5 with peer 7e6d75f768764435ab5f0b63159ca43a
I20240502 08:05:32.255028 18603 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:32.255324 18603 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap complete.
I20240502 08:05:32.255501 18603 ts_tablet_manager.cc:1367] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: Time spent bootstrapping tablet: real 0.023s user 0.020s sys 0.001s
I20240502 08:05:32.255903 18603 raft_consensus.cc:348] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.256078 18603 raft_consensus.cc:729] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e6d75f768764435ab5f0b63159ca43a, State: Initialized, Role: LEARNER
I20240502 08:05:32.256227 18603 consensus_queue.cc:260] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:32.256564 18603 ts_tablet_manager.cc:1398] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 08:05:32.257544 17387 tablet_copy_service.cc:342] P d8da072ac9514d3097eb441c8ee68428: Request end of tablet copy session 7e6d75f768764435ab5f0b63159ca43a-c0180a09549b4556addb20ce1f8d4e63 received from {username='slave'} at 127.16.36.196:37419
I20240502 08:05:32.257630 17387 tablet_copy_service.cc:433] P d8da072ac9514d3097eb441c8ee68428: ending tablet copy session 7e6d75f768764435ab5f0b63159ca43a-c0180a09549b4556addb20ce1f8d4e63 on tablet c0180a09549b4556addb20ce1f8d4e63 with peer 7e6d75f768764435ab5f0b63159ca43a
I20240502 08:05:32.612848 18270 raft_consensus.cc:1203] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:32.668273 18270 raft_consensus.cc:1203] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:32.670337 18139 raft_consensus.cc:1203] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:32.673278 18139 raft_consensus.cc:1203] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:32.772089 18139 raft_consensus.cc:1203] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:32.790819 18270 raft_consensus.cc:1203] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:32.838464 18270 raft_consensus.cc:1203] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:33.051493 18708 raft_consensus.cc:1050] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: attempting to promote NON_VOTER 7e6d75f768764435ab5f0b63159ca43a to VOTER
I20240502 08:05:33.052075 18708 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.053735 17367 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.053804 17235 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.053793 18270 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.054251 18784 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.054455 18708 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.054689 18707 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.056959 18707 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.059002 17367 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.059269 17235 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.059432 17084 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:33.059780 18270 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.062103 18710 raft_consensus.cc:1050] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458: attempting to promote NON_VOTER 7e6d75f768764435ab5f0b63159ca43a to VOTER
I20240502 08:05:33.062621 18710 consensus_queue.cc:237] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.064147 17499 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.064414 18270 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.064806 18793 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.065225 17367 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.065320 18709 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.065613 18710 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.066886 17497 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.067418 18710 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.067848 18270 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.068264 17235 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.068493 18269 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.068929 18521 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.069372 18784 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.069828 17367 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.070115 17497 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.070998 17088 catalog_manager.cc:5484] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:33.071308 18521 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.072821 18270 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.073323 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efbde964902249719b24cf743249ec6f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.073868 17235 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.075408 17234 consensus_queue.cc:237] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.076076 17091 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a reported cstate change: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.077497 18270 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.077623 17367 raft_consensus.cc:1261] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.077965 18793 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.078174 18709 consensus_queue.cc:1035] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.078797 17346 tablet_service.cc:1514] Processing DeleteTablet for tablet efbde964902249719b24cf743249ec6f with delete_type TABLET_DATA_TOMBSTONED (TS d8da072ac9514d3097eb441c8ee68428 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58026
I20240502 08:05:33.079427 18797 tablet_replica.cc:330] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: stopping tablet replica
I20240502 08:05:33.079753 18797 raft_consensus.cc:2229] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.080034 18797 raft_consensus.cc:2258] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.080130 18464 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.080551 17367 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.080623 18270 raft_consensus.cc:2932] T c095f330e9d941108ceb48b65c4e4327 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.082020 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c095f330e9d941108ceb48b65c4e4327 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.082561 18797 ts_tablet_manager.cc:1875] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.082582 17088 catalog_manager.cc:5484] T c095f330e9d941108ceb48b65c4e4327 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.084833 18797 ts_tablet_manager.cc:1888] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.084940 18797 log.cc:1196] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/efbde964902249719b24cf743249ec6f
I20240502 08:05:33.085582 17078 catalog_manager.cc:4830] TS d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347): tablet efbde964902249719b24cf743249ec6f (table rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]) successfully deleted
I20240502 08:05:33.085585 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet c095f330e9d941108ceb48b65c4e4327 with delete_type TABLET_DATA_TOMBSTONED (TS 1256e2e5163d4240bd7dd791512ffa5c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42336
I20240502 08:05:33.086032 18798 tablet_replica.cc:330] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: stopping tablet replica
I20240502 08:05:33.086318 18798 raft_consensus.cc:2229] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.086520 18798 raft_consensus.cc:2258] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.087488 18798 ts_tablet_manager.cc:1875] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.089356 18798 ts_tablet_manager.cc:1888] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.089432 18798 log.cc:1196] T c095f330e9d941108ceb48b65c4e4327 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/c095f330e9d941108ceb48b65c4e4327
I20240502 08:05:33.089924 17078 catalog_manager.cc:4830] TS 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337): tablet c095f330e9d941108ceb48b65c4e4327 (table rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]) successfully deleted
I20240502 08:05:33.146271 18711 raft_consensus.cc:1050] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428: attempting to promote NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d to VOTER
I20240502 08:05:33.146827 18711 consensus_queue.cc:237] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:33.148244 17497 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.148247 18139 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.148319 17234 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.148726 18712 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.148890 18711 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.149121 18800 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.151038 18712 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.151479 17497 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.152493 18139 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.153046 17234 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.154174 17091 catalog_manager.cc:5484] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.157112 18707 raft_consensus.cc:1050] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c: attempting to promote NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d to VOTER
I20240502 08:05:33.157506 18707 consensus_queue.cc:237] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:33.158810 17367 consensus_queue.cc:237] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:33.159056 18139 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.159066 17365 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.159276 17234 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.159432 18784 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.159646 18708 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.160252 18139 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.160288 17497 raft_consensus.cc:1261] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.160367 18708 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.160699 18800 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.160920 18711 consensus_queue.cc:1035] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.162396 18499 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.162886 18138 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.162878 17497 raft_consensus.cc:2932] T 4da8a9287c5b48ac9e990ecaac40fc9e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.164810 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4da8a9287c5b48ac9e990ecaac40fc9e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.165714 17088 catalog_manager.cc:5484] T 4da8a9287c5b48ac9e990ecaac40fc9e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:33.172917 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da8a9287c5b48ac9e990ecaac40fc9e with delete_type TABLET_DATA_TOMBSTONED (TS d67b9894d6dc4495ad18c22ee4a0d458 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33218
I20240502 08:05:33.172928 18708 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.173810 17234 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.175098 17367 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.175537 17092 catalog_manager.cc:5484] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:33.177253 18139 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.182010 17497 consensus_queue.cc:237] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:33.183142 18139 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.183192 17234 raft_consensus.cc:1261] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.183219 18807 tablet_replica.cc:330] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: stopping tablet replica
I20240502 08:05:33.183540 18807 raft_consensus.cc:2229] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.183614 18784 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.183768 18807 raft_consensus.cc:2258] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.183815 18708 consensus_queue.cc:1035] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.185014 18807 ts_tablet_manager.cc:1875] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.185954 18521 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.187580 18807 ts_tablet_manager.cc:1888] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.187741 18807 log.cc:1196] T 4da8a9287c5b48ac9e990ecaac40fc9e P d67b9894d6dc4495ad18c22ee4a0d458: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/4da8a9287c5b48ac9e990ecaac40fc9e
I20240502 08:05:33.188274 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4735728a9541425c8482ed7fb17bd483 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.188503 18707 raft_consensus.cc:1050] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c: attempting to promote NON_VOTER 7e6d75f768764435ab5f0b63159ca43a to VOTER
I20240502 08:05:33.188485 17078 catalog_manager.cc:4830] TS d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): tablet 4da8a9287c5b48ac9e990ecaac40fc9e (table rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]) successfully deleted
I20240502 08:05:33.188804 17234 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.188900 18707 consensus_queue.cc:237] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.190855 17367 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.190706 17084 catalog_manager.cc:5484] T 4735728a9541425c8482ed7fb17bd483 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:33.190912 18139 raft_consensus.cc:2932] T 4735728a9541425c8482ed7fb17bd483 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.191303 17234 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.191689 18270 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.192778 18712 raft_consensus.cc:1050] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428: attempting to promote NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d to VOTER
I20240502 08:05:33.193182 18712 consensus_queue.cc:237] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:33.193967 17346 tablet_service.cc:1514] Processing DeleteTablet for tablet 4735728a9541425c8482ed7fb17bd483 with delete_type TABLET_DATA_TOMBSTONED (TS d8da072ac9514d3097eb441c8ee68428 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58026
I20240502 08:05:33.194183 18797 tablet_replica.cc:330] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: stopping tablet replica
I20240502 08:05:33.194314 18797 raft_consensus.cc:2229] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.194905 18797 raft_consensus.cc:2258] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.194944 18138 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.195004 17497 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.195504 17234 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.195860 18797 ts_tablet_manager.cc:1875] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.195866 18711 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.196260 18711 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.196749 18707 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.197104 18707 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.197396 18707 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.198169 18711 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.201553 18797 ts_tablet_manager.cc:1888] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.201695 18797 log.cc:1196] T 4735728a9541425c8482ed7fb17bd483 P d8da072ac9514d3097eb441c8ee68428: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/4735728a9541425c8482ed7fb17bd483
I20240502 08:05:33.202145 18521 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.202632 18800 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.203063 18270 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.203104 17497 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.205453 17092 catalog_manager.cc:5484] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:33.205540 17234 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.206734 18138 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.207408 17078 catalog_manager.cc:4830] TS d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347): tablet 4735728a9541425c8482ed7fb17bd483 (table rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]) successfully deleted
I20240502 08:05:33.207703 17365 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.207890 17084 catalog_manager.cc:5484] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:33.209471 17234 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.210261 17497 consensus_queue.cc:237] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.211757 18270 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.212141 17234 raft_consensus.cc:1261] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.212608 18707 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.212934 18784 consensus_queue.cc:1035] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.215421 17365 consensus_queue.cc:237] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:33.215962 18707 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.216575 18138 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.216997 17234 raft_consensus.cc:1261] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.217540 18711 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.217746 18800 consensus_queue.cc:1035] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.217820 17235 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.219993 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84a7fcaa3ee34406827fcbabfda1aed5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.219962 18800 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.219949 18270 raft_consensus.cc:2932] T 84a7fcaa3ee34406827fcbabfda1aed5 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.220719 17091 catalog_manager.cc:5484] T 84a7fcaa3ee34406827fcbabfda1aed5 P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index 5 to 6, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.221817 18138 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.223846 17346 tablet_service.cc:1514] Processing DeleteTablet for tablet 84a7fcaa3ee34406827fcbabfda1aed5 with delete_type TABLET_DATA_TOMBSTONED (TS d8da072ac9514d3097eb441c8ee68428 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58026
I20240502 08:05:33.224040 18797 tablet_replica.cc:330] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: stopping tablet replica
I20240502 08:05:33.224154 18797 raft_consensus.cc:2229] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.224321 18797 raft_consensus.cc:2258] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.224404 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a76ee285ba8645a88251bd351c4e4146 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.225385 18797 ts_tablet_manager.cc:1875] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.227701 18797 ts_tablet_manager.cc:1888] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.227782 18797 log.cc:1196] T 84a7fcaa3ee34406827fcbabfda1aed5 P d8da072ac9514d3097eb441c8ee68428: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/84a7fcaa3ee34406827fcbabfda1aed5
I20240502 08:05:33.228335 17078 catalog_manager.cc:4830] TS d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347): tablet 84a7fcaa3ee34406827fcbabfda1aed5 (table rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]) successfully deleted
I20240502 08:05:33.228634 17088 catalog_manager.cc:5484] T a76ee285ba8645a88251bd351c4e4146 P 93ae1f0181ba4eae9801831b0e037f6d reported cstate change: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.231505 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet a76ee285ba8645a88251bd351c4e4146 with delete_type TABLET_DATA_TOMBSTONED (TS 1256e2e5163d4240bd7dd791512ffa5c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42336
I20240502 08:05:33.231727 18798 tablet_replica.cc:330] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: stopping tablet replica
I20240502 08:05:33.231896 18798 raft_consensus.cc:2229] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.232066 18798 raft_consensus.cc:2258] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.232180 17234 raft_consensus.cc:2932] T a76ee285ba8645a88251bd351c4e4146 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:33.233286 18798 ts_tablet_manager.cc:1875] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.235536 18798 ts_tablet_manager.cc:1888] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.235592 18798 log.cc:1196] T a76ee285ba8645a88251bd351c4e4146 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/a76ee285ba8645a88251bd351c4e4146
I20240502 08:05:33.236091 17078 catalog_manager.cc:4830] TS 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337): tablet a76ee285ba8645a88251bd351c4e4146 (table rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]) successfully deleted
I20240502 08:05:33.264206 18800 raft_consensus.cc:1050] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: attempting to promote NON_VOTER 7e6d75f768764435ab5f0b63159ca43a to VOTER
I20240502 08:05:33.264639 18800 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.266048 18270 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.266156 17234 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.266166 17497 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:33.266579 18499 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.266834 18711 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.266995 18712 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:33.269316 18712 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.269841 17497 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.270103 17234 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.271869 18270 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.272337 17091 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:33.276286 17365 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:33.277702 18270 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.277797 17497 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:33.278185 18800 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.278369 18712 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:33.279922 18711 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.280388 18270 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.280412 17497 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.282580 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0180a09549b4556addb20ce1f8d4e63 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:33.282783 17092 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 5 to 6, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:33.285318 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet c0180a09549b4556addb20ce1f8d4e63 with delete_type TABLET_DATA_TOMBSTONED (TS d67b9894d6dc4495ad18c22ee4a0d458 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33218
I20240502 08:05:33.285580 18807 tablet_replica.cc:330] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: stopping tablet replica
I20240502 08:05:33.285708 18807 raft_consensus.cc:2229] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:33.285833 18807 raft_consensus.cc:2258] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:33.286399 18807 ts_tablet_manager.cc:1875] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:33.288108 18807 ts_tablet_manager.cc:1888] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:33.288173 18807 log.cc:1196] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0180a09549b4556addb20ce1f8d4e63
I20240502 08:05:33.288695 17078 catalog_manager.cc:4830] TS d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): tablet c0180a09549b4556addb20ce1f8d4e63 (table rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]) successfully deleted
I20240502 08:05:33.521347 17477 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:33.522941 17214 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:33.533293 18119 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:33.535660 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:33.536087 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:33.571942 17497 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:33.573488 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:05:33.573537 17433 consensus_peers.cc:485] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c -> Peer 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867): Couldn't send request to peer 93ae1f0181ba4eae9801831b0e037f6d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df7c076e0edb4c9089c58c4a3d55eb83. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:33.573944 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:33.573942 18784 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:33.574309 18784 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:33.575778 18521 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:33.576253 17234 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:33.576423 17365 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:33.579383 17084 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index -1 to 4, NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) added. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } } }
I20240502 08:05:33.586334 17234 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:33.587817 17365 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:33.587841 17497 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:05:33.587881 17169 consensus_peers.cc:485] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 -> Peer 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989): Couldn't send request to peer 7e6d75f768764435ab5f0b63159ca43a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38d81695cf9d4c1faf2b67580d44480e. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:33.588335 18911 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:33.588568 18793 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:33.590507 18910 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:33.590984 17365 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:33.590974 17497 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } } }
I20240502 08:05:33.593127 17088 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index -1 to 4, NON_VOTER 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:33.997403 18934 ts_tablet_manager.cc:902] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Initiating tablet copy from peer 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:33.997951 18934 tablet_copy_client.cc:308] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:41337
I20240502 08:05:33.998445 17518 tablet_copy_service.cc:140] P 1256e2e5163d4240bd7dd791512ffa5c: Received BeginTabletCopySession request for tablet df7c076e0edb4c9089c58c4a3d55eb83 from peer 93ae1f0181ba4eae9801831b0e037f6d ({username='slave'} at 127.16.36.197:33181)
I20240502 08:05:33.998596 17518 tablet_copy_service.cc:161] P 1256e2e5163d4240bd7dd791512ffa5c: Beginning new tablet copy session on tablet df7c076e0edb4c9089c58c4a3d55eb83 from peer 93ae1f0181ba4eae9801831b0e037f6d at {username='slave'} at 127.16.36.197:33181: session id = 93ae1f0181ba4eae9801831b0e037f6d-df7c076e0edb4c9089c58c4a3d55eb83
I20240502 08:05:33.999629 17518 tablet_copy_source_session.cc:205] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:34.000638 18934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7c076e0edb4c9089c58c4a3d55eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:34.003082 18934 tablet_copy_client.cc:783] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:34.003221 18934 tablet_copy_client.cc:647] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:34.004541 18934 tablet_copy_client.cc:523] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:34.005584 18934 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap starting.
I20240502 08:05:34.026947 18934 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:34.027271 18934 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Bootstrap complete.
I20240502 08:05:34.027465 18934 ts_tablet_manager.cc:1367] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Time spent bootstrapping tablet: real 0.022s user 0.020s sys 0.000s
I20240502 08:05:34.027967 18934 raft_consensus.cc:348] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:34.028105 18934 raft_consensus.cc:729] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 93ae1f0181ba4eae9801831b0e037f6d, State: Initialized, Role: LEARNER
I20240502 08:05:34.028234 18934 consensus_queue.cc:260] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: NON_VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: true } }
I20240502 08:05:34.028673 18934 ts_tablet_manager.cc:1398] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 08:05:34.029199 17518 tablet_copy_service.cc:342] P 1256e2e5163d4240bd7dd791512ffa5c: Request end of tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-df7c076e0edb4c9089c58c4a3d55eb83 received from {username='slave'} at 127.16.36.197:33181
I20240502 08:05:34.029276 17518 tablet_copy_service.cc:433] P 1256e2e5163d4240bd7dd791512ffa5c: ending tablet copy session 93ae1f0181ba4eae9801831b0e037f6d-df7c076e0edb4c9089c58c4a3d55eb83 on tablet df7c076e0edb4c9089c58c4a3d55eb83 with peer 93ae1f0181ba4eae9801831b0e037f6d
I20240502 08:05:34.081957 18951 ts_tablet_manager.cc:902] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: Initiating tablet copy from peer d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:34.082410 18951 tablet_copy_client.cc:308] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:46425
I20240502 08:05:34.082988 17255 tablet_copy_service.cc:140] P d67b9894d6dc4495ad18c22ee4a0d458: Received BeginTabletCopySession request for tablet 38d81695cf9d4c1faf2b67580d44480e from peer 7e6d75f768764435ab5f0b63159ca43a ({username='slave'} at 127.16.36.196:46541)
I20240502 08:05:34.083110 17255 tablet_copy_service.cc:161] P d67b9894d6dc4495ad18c22ee4a0d458: Beginning new tablet copy session on tablet 38d81695cf9d4c1faf2b67580d44480e from peer 7e6d75f768764435ab5f0b63159ca43a at {username='slave'} at 127.16.36.196:46541: session id = 7e6d75f768764435ab5f0b63159ca43a-38d81695cf9d4c1faf2b67580d44480e
I20240502 08:05:34.083894 17255 tablet_copy_source_session.cc:205] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:34.084851 18951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d81695cf9d4c1faf2b67580d44480e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:34.086592 18951 tablet_copy_client.cc:783] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:34.086738 18951 tablet_copy_client.cc:647] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:34.087872 18951 tablet_copy_client.cc:523] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:34.088932 18951 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap starting.
I20240502 08:05:34.106482 18951 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:34.106797 18951 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: Bootstrap complete.
I20240502 08:05:34.106983 18951 ts_tablet_manager.cc:1367] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.003s
I20240502 08:05:34.107386 18951 raft_consensus.cc:348] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:34.107497 18951 raft_consensus.cc:729] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7e6d75f768764435ab5f0b63159ca43a, State: Initialized, Role: LEARNER
I20240502 08:05:34.107597 18951 consensus_queue.cc:260] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: true } }
I20240502 08:05:34.107942 18951 ts_tablet_manager.cc:1398] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a: Time spent starting tablet: real 0.001s user 0.003s sys 0.001s
I20240502 08:05:34.108600 17255 tablet_copy_service.cc:342] P d67b9894d6dc4495ad18c22ee4a0d458: Request end of tablet copy session 7e6d75f768764435ab5f0b63159ca43a-38d81695cf9d4c1faf2b67580d44480e received from {username='slave'} at 127.16.36.196:46541
I20240502 08:05:34.108687 17255 tablet_copy_service.cc:433] P d67b9894d6dc4495ad18c22ee4a0d458: ending tablet copy session 7e6d75f768764435ab5f0b63159ca43a-38d81695cf9d4c1faf2b67580d44480e on tablet 38d81695cf9d4c1faf2b67580d44480e with peer 7e6d75f768764435ab5f0b63159ca43a
I20240502 08:05:34.475981 18138 raft_consensus.cc:1203] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:34.589623 18270 raft_consensus.cc:1203] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:34.970158 18979 raft_consensus.cc:1050] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: attempting to promote NON_VOTER 93ae1f0181ba4eae9801831b0e037f6d to VOTER
I20240502 08:05:34.970597 18979 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:34.971972 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:34.971972 18138 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 LEARNER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:34.972064 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:34.972401 19003 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:34.972620 18708 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:34.972945 18979 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:34.974992 18979 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:34.975450 18138 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:34.975669 17365 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:34.977815 17234 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:34.978181 17084 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d reported cstate change: config changed from index 4 to 5, 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:35.020714 18981 raft_consensus.cc:1050] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: attempting to promote NON_VOTER 7e6d75f768764435ab5f0b63159ca43a to VOTER
I20240502 08:05:35.021175 18981 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:35.022864 18270 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 1 LEARNER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:35.022902 17365 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:35.022933 17497 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d67b9894d6dc4495ad18c22ee4a0d458: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:35.023344 18793 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:35.023558 18980 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:35.023880 19013 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:35.025955 18980 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:35.026463 17497 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:35.026533 18270 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:35.028982 17092 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 reported cstate change: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:35.029844 17365 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:35.101454 17234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e"
dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50040
I20240502 08:05:35.101642 17234 raft_consensus.cc:595] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:35.112216 17497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83"
dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57646
I20240502 08:05:35.112399 17497 raft_consensus.cc:595] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:35.325559 17234 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e"
dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50058
I20240502 08:05:35.325711 17234 raft_consensus.cc:595] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:35.337129 17497 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83"
dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57664
I20240502 08:05:35.337275 17497 raft_consensus.cc:595] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:35.437227 19013 raft_consensus.cc:979] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: : Instructing follower d8da072ac9514d3097eb441c8ee68428 to start an election
I20240502 08:05:35.437377 19013 raft_consensus.cc:1067] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Signalling peer d8da072ac9514d3097eb441c8ee68428 to start an election
I20240502 08:05:35.437996 17365 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e"
dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
from {username='slave'} at 127.16.36.193:42791
I20240502 08:05:35.438191 17365 raft_consensus.cc:482] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:35.438270 17365 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.439285 17365 raft_consensus.cc:504] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:35.439673 17365 leader_election.cc:288] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 2 election: Requested vote from peers d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337), 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196:38989)
I20240502 08:05:35.440127 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:35.440172 18270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7e6d75f768764435ab5f0b63159ca43a"
I20240502 08:05:35.440182 17497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38d81695cf9d4c1faf2b67580d44480e" candidate_uuid: "d8da072ac9514d3097eb441c8ee68428" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:35.440266 17234 raft_consensus.cc:3032] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:35.440306 17497 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.440312 17234 raft_consensus.cc:729] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Running, Role: LEADER
I20240502 08:05:35.440343 18270 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.440541 17234 consensus_queue.cc:260] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:35.440781 17234 raft_consensus.cc:3037] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.441149 17497 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 2.
I20240502 08:05:35.441481 18270 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 2.
I20240502 08:05:35.441797 17234 raft_consensus.cc:2454] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8da072ac9514d3097eb441c8ee68428 in term 2.
I20240502 08:05:35.441874 17301 leader_election.cc:302] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, 7e6d75f768764435ab5f0b63159ca43a, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:35.442121 18711 raft_consensus.cc:2781] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:35.442268 18711 raft_consensus.cc:686] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 LEADER]: Becoming Leader. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Running, Role: LEADER
I20240502 08:05:35.442396 18711 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:35.444164 17088 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: term changed from 1 to 2, leader changed from d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) to d8da072ac9514d3097eb441c8ee68428 (127.16.36.194). New cstate: current_term: 2 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:35.454711 19003 raft_consensus.cc:979] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: : Instructing follower 93ae1f0181ba4eae9801831b0e037f6d to start an election
I20240502 08:05:35.454905 19003 raft_consensus.cc:1067] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Signalling peer 93ae1f0181ba4eae9801831b0e037f6d to start an election
I20240502 08:05:35.455526 18138 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83"
dest_uuid: "93ae1f0181ba4eae9801831b0e037f6d"
from {username='slave'} at 127.16.36.195:41359
I20240502 08:05:35.455724 18138 raft_consensus.cc:482] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:35.455821 18138 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.456702 18138 raft_consensus.cc:504] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:35.457724 18138 leader_election.cc:288] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [CANDIDATE]: Term 2 election: Requested vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:35.458104 17365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "93ae1f0181ba4eae9801831b0e037f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:35.458252 17365 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.458485 17497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "93ae1f0181ba4eae9801831b0e037f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:35.458626 17497 raft_consensus.cc:3032] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:35.458674 17497 raft_consensus.cc:729] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Running, Role: LEADER
I20240502 08:05:35.458833 17497 consensus_queue.cc:260] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:35.459038 17497 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.459455 17365 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93ae1f0181ba4eae9801831b0e037f6d in term 2.
I20240502 08:05:35.459988 17497 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93ae1f0181ba4eae9801831b0e037f6d in term 2.
I20240502 08:05:35.460314 18073 leader_election.cc:302] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, 93ae1f0181ba4eae9801831b0e037f6d, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:35.460608 19007 raft_consensus.cc:2781] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:35.460810 19007 raft_consensus.cc:686] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 LEADER]: Becoming Leader. State: Replica: 93ae1f0181ba4eae9801831b0e037f6d, State: Running, Role: LEADER
I20240502 08:05:35.461117 19007 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:35.461910 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df7c076e0edb4c9089c58c4a3d55eb83" candidate_uuid: "93ae1f0181ba4eae9801831b0e037f6d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:35.462044 17234 raft_consensus.cc:3037] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:35.462935 17234 raft_consensus.cc:2454] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93ae1f0181ba4eae9801831b0e037f6d in term 2.
I20240502 08:05:35.463716 17088 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d reported cstate change: term changed from 1 to 2, leader changed from 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) to 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197). New cstate: current_term: 2 leader_uuid: "93ae1f0181ba4eae9801831b0e037f6d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:35.842200 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:35.842864 19007 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:35.862443 17497 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:35.863022 19007 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:35.866268 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:35.867010 19059 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:35.908370 17234 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:35.908885 18711 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:35.993686 17497 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:35.994143 18711 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:35.996912 18270 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:35.997388 19098 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:36.401604 18138 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } }
I20240502 08:05:36.402890 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:36.402894 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:36.403319 19116 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.403549 19117 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.405700 19116 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:36.407109 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df7c076e0edb4c9089c58c4a3d55eb83 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:36.407626 17365 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:36.407711 17234 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } }
I20240502 08:05:36.407979 17092 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d reported cstate change: config changed from index 5 to 7, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "93ae1f0181ba4eae9801831b0e037f6d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:36.413398 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet df7c076e0edb4c9089c58c4a3d55eb83 with delete_type TABLET_DATA_TOMBSTONED (TS 1256e2e5163d4240bd7dd791512ffa5c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42336
I20240502 08:05:36.414232 19132 tablet_replica.cc:330] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: stopping tablet replica
I20240502 08:05:36.414444 19132 raft_consensus.cc:2229] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:36.414613 19132 raft_consensus.cc:2258] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:36.415341 19132 ts_tablet_manager.cc:1875] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:36.417706 19132 ts_tablet_manager.cc:1888] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:05:36.417778 19132 log.cc:1196] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/df7c076e0edb4c9089c58c4a3d55eb83
I20240502 08:05:36.418334 17078 catalog_manager.cc:4830] TS 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337): tablet df7c076e0edb4c9089c58c4a3d55eb83 (table rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]) successfully deleted
I20240502 08:05:36.471637 17365 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } }
I20240502 08:05:36.472806 18270 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:36.472915 17499 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:36.473218 18711 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.473508 19098 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.475078 18711 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:36.475572 18270 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:36.475797 17499 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:36.476918 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38d81695cf9d4c1faf2b67580d44480e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:36.478406 17088 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a reported cstate change: config changed from index 5 to 7, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } }
I20240502 08:05:36.480754 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d81695cf9d4c1faf2b67580d44480e with delete_type TABLET_DATA_TOMBSTONED (TS d67b9894d6dc4495ad18c22ee4a0d458 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33218
I20240502 08:05:36.481141 19133 tablet_replica.cc:330] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: stopping tablet replica
I20240502 08:05:36.481259 19133 raft_consensus.cc:2229] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:36.481352 19133 raft_consensus.cc:2258] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:36.481931 19133 ts_tablet_manager.cc:1875] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:36.483672 19133 ts_tablet_manager.cc:1888] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 08:05:36.483748 19133 log.cc:1196] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/38d81695cf9d4c1faf2b67580d44480e
I20240502 08:05:36.484272 17078 catalog_manager.cc:4830] TS d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): tablet 38d81695cf9d4c1faf2b67580d44480e (table rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]) successfully deleted
I20240502 08:05:36.677546 18119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:36.681268 17346 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:36.681366 17477 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:36.681907 17214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:36.682323 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:36.773938 17477 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:36.775707 18119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:36.776293 18250 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:36.778604 17214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:36.778952 17346 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 08:05:36.820417 17365 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } }
I20240502 08:05:36.822067 18270 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:36.822070 17499 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 08:05:36.822396 17301 consensus_peers.cc:485] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 -> Peer d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): Couldn't send request to peer d67b9894d6dc4495ad18c22ee4a0d458. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:36.822579 19097 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:36.822775 18711 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:36.824576 19098 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } } }
I20240502 08:05:36.826352 17499 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } } }
I20240502 08:05:36.826351 18270 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } } }
I20240502 08:05:36.829303 17091 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 7 to 8, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New cstate: current_term: 2 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.836205 17365 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } }
I20240502 08:05:36.837700 18270 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:36.837705 17499 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:05:36.837993 17301 consensus_peers.cc:485] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 -> Peer d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): Couldn't send request to peer d67b9894d6dc4495ad18c22ee4a0d458. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:36.838214 19098 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.838435 19097 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.840164 19098 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } } }
I20240502 08:05:36.840634 18270 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } } }
I20240502 08:05:36.840914 17499 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } } }
I20240502 08:05:36.842695 17092 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 6 to 7, NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) added. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.850999 18138 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } }
I20240502 08:05:36.852485 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:36.852546 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240502 08:05:36.852758 18073 consensus_peers.cc:485] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d -> Peer 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337): Couldn't send request to peer 1256e2e5163d4240bd7dd791512ffa5c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:36.853004 19083 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:36.853188 19117 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:36.854835 19116 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } } }
I20240502 08:05:36.855294 17234 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } } }
I20240502 08:05:36.855399 17365 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } } }
I20240502 08:05:36.858248 17091 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d reported cstate change: config changed from index 7 to 8, NON_VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) added. New cstate: current_term: 2 leader_uuid: "93ae1f0181ba4eae9801831b0e037f6d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:36.865746 17499 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } }
I20240502 08:05:36.867194 18270 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:36.867223 17234 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 08:05:36.867555 17433 consensus_peers.cc:485] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c -> Peer d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347): Couldn't send request to peer d8da072ac9514d3097eb441c8ee68428. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:36.867765 19235 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.867973 19003 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:36.869695 19235 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } } }
I20240502 08:05:36.870119 18270 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } } }
I20240502 08:05:36.870458 17234 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } } }
I20240502 08:05:36.873066 17092 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a reported cstate change: config changed from index 6 to 7, NON_VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) added. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } } }
I20240502 08:05:37.230636 19278 ts_tablet_manager.cc:902] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Initiating tablet copy from peer d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:37.231343 19278 tablet_copy_client.cc:308] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:35347
I20240502 08:05:37.231958 17387 tablet_copy_service.cc:140] P d8da072ac9514d3097eb441c8ee68428: Received BeginTabletCopySession request for tablet c0180a09549b4556addb20ce1f8d4e63 from peer d67b9894d6dc4495ad18c22ee4a0d458 ({username='slave'} at 127.16.36.193:42791)
I20240502 08:05:37.232072 17387 tablet_copy_service.cc:161] P d8da072ac9514d3097eb441c8ee68428: Beginning new tablet copy session on tablet c0180a09549b4556addb20ce1f8d4e63 from peer d67b9894d6dc4495ad18c22ee4a0d458 at {username='slave'} at 127.16.36.193:42791: session id = d67b9894d6dc4495ad18c22ee4a0d458-c0180a09549b4556addb20ce1f8d4e63
I20240502 08:05:37.232877 17387 tablet_copy_source_session.cc:205] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:37.233937 19278 ts_tablet_manager.cc:1875] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:37.237187 19278 ts_tablet_manager.cc:1888] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:37.237377 19278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0180a09549b4556addb20ce1f8d4e63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:37.238384 19278 tablet_copy_client.cc:783] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:37.238565 19278 tablet_copy_client.cc:647] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:37.240026 19278 tablet_copy_client.cc:523] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:37.241470 19278 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:37.260442 19278 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:37.260823 19278 tablet_bootstrap.cc:492] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap complete.
I20240502 08:05:37.260980 19278 ts_tablet_manager.cc:1367] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.020s user 0.020s sys 0.000s
I20240502 08:05:37.261368 19278 raft_consensus.cc:348] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } }
I20240502 08:05:37.261480 19278 raft_consensus.cc:729] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: LEARNER
I20240502 08:05:37.261554 19278 consensus_queue.cc:260] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } }
I20240502 08:05:37.261910 19278 ts_tablet_manager.cc:1398] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:37.262583 17387 tablet_copy_service.cc:342] P d8da072ac9514d3097eb441c8ee68428: Request end of tablet copy session d67b9894d6dc4495ad18c22ee4a0d458-c0180a09549b4556addb20ce1f8d4e63 received from {username='slave'} at 127.16.36.193:42791
I20240502 08:05:37.262668 17387 tablet_copy_service.cc:433] P d8da072ac9514d3097eb441c8ee68428: ending tablet copy session d67b9894d6dc4495ad18c22ee4a0d458-c0180a09549b4556addb20ce1f8d4e63 on tablet c0180a09549b4556addb20ce1f8d4e63 with peer d67b9894d6dc4495ad18c22ee4a0d458
I20240502 08:05:37.306866 19284 ts_tablet_manager.cc:902] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Initiating tablet copy from peer 93ae1f0181ba4eae9801831b0e037f6d (127.16.36.197:44867)
I20240502 08:05:37.307725 19284 tablet_copy_client.cc:308] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.197:44867
I20240502 08:05:37.308636 18159 tablet_copy_service.cc:140] P 93ae1f0181ba4eae9801831b0e037f6d: Received BeginTabletCopySession request for tablet df7c076e0edb4c9089c58c4a3d55eb83 from peer 1256e2e5163d4240bd7dd791512ffa5c ({username='slave'} at 127.16.36.195:41359)
I20240502 08:05:37.308827 18159 tablet_copy_service.cc:161] P 93ae1f0181ba4eae9801831b0e037f6d: Beginning new tablet copy session on tablet df7c076e0edb4c9089c58c4a3d55eb83 from peer 1256e2e5163d4240bd7dd791512ffa5c at {username='slave'} at 127.16.36.195:41359: session id = 1256e2e5163d4240bd7dd791512ffa5c-df7c076e0edb4c9089c58c4a3d55eb83
I20240502 08:05:37.309983 18159 tablet_copy_source_session.cc:205] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:37.311183 19284 ts_tablet_manager.cc:1875] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:37.313061 19284 ts_tablet_manager.cc:1888] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 08:05:37.313257 19284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df7c076e0edb4c9089c58c4a3d55eb83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:37.314267 19284 tablet_copy_client.cc:783] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:37.314421 19284 tablet_copy_client.cc:647] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:37.316118 19284 tablet_copy_client.cc:523] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:37.317620 19284 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap starting.
I20240502 08:05:37.341733 19284 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:37.342178 19284 tablet_bootstrap.cc:492] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Bootstrap complete.
I20240502 08:05:37.342386 19284 ts_tablet_manager.cc:1367] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent bootstrapping tablet: real 0.025s user 0.025s sys 0.000s
I20240502 08:05:37.342887 19284 raft_consensus.cc:348] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } }
I20240502 08:05:37.343034 19284 raft_consensus.cc:729] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1256e2e5163d4240bd7dd791512ffa5c, State: Initialized, Role: LEARNER
I20240502 08:05:37.343153 19284 consensus_queue.cc:260] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: true } }
I20240502 08:05:37.343554 19284 ts_tablet_manager.cc:1398] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:37.344410 18159 tablet_copy_service.cc:342] P 93ae1f0181ba4eae9801831b0e037f6d: Request end of tablet copy session 1256e2e5163d4240bd7dd791512ffa5c-df7c076e0edb4c9089c58c4a3d55eb83 received from {username='slave'} at 127.16.36.195:41359
I20240502 08:05:37.344530 18159 tablet_copy_service.cc:433] P 93ae1f0181ba4eae9801831b0e037f6d: ending tablet copy session 1256e2e5163d4240bd7dd791512ffa5c-df7c076e0edb4c9089c58c4a3d55eb83 on tablet df7c076e0edb4c9089c58c4a3d55eb83 with peer 1256e2e5163d4240bd7dd791512ffa5c
I20240502 08:05:37.353922 19278 ts_tablet_manager.cc:902] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Initiating tablet copy from peer d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347)
I20240502 08:05:37.354429 19278 tablet_copy_client.cc:308] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:35347
I20240502 08:05:37.354925 17387 tablet_copy_service.cc:140] P d8da072ac9514d3097eb441c8ee68428: Received BeginTabletCopySession request for tablet 38d81695cf9d4c1faf2b67580d44480e from peer d67b9894d6dc4495ad18c22ee4a0d458 ({username='slave'} at 127.16.36.193:42791)
I20240502 08:05:37.355033 17387 tablet_copy_service.cc:161] P d8da072ac9514d3097eb441c8ee68428: Beginning new tablet copy session on tablet 38d81695cf9d4c1faf2b67580d44480e from peer d67b9894d6dc4495ad18c22ee4a0d458 at {username='slave'} at 127.16.36.193:42791: session id = d67b9894d6dc4495ad18c22ee4a0d458-38d81695cf9d4c1faf2b67580d44480e
I20240502 08:05:37.355825 17387 tablet_copy_source_session.cc:205] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:37.356581 19278 ts_tablet_manager.cc:1875] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:37.358881 19278 ts_tablet_manager.cc:1888] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.6
I20240502 08:05:37.359058 19278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38d81695cf9d4c1faf2b67580d44480e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:37.360093 19278 tablet_copy_client.cc:783] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:37.360235 19278 tablet_copy_client.cc:647] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:37.361466 19278 tablet_copy_client.cc:523] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:37.362665 19278 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap starting.
I20240502 08:05:37.382826 19278 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:37.383177 19278 tablet_bootstrap.cc:492] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Bootstrap complete.
I20240502 08:05:37.383334 19278 ts_tablet_manager.cc:1367] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Time spent bootstrapping tablet: real 0.021s user 0.019s sys 0.000s
I20240502 08:05:37.383761 19278 raft_consensus.cc:348] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } }
I20240502 08:05:37.383873 19278 raft_consensus.cc:729] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d67b9894d6dc4495ad18c22ee4a0d458, State: Initialized, Role: LEARNER
I20240502 08:05:37.383965 19278 consensus_queue.cc:260] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: NON_VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: true } }
I20240502 08:05:37.384310 19278 ts_tablet_manager.cc:1398] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:37.385031 17387 tablet_copy_service.cc:342] P d8da072ac9514d3097eb441c8ee68428: Request end of tablet copy session d67b9894d6dc4495ad18c22ee4a0d458-38d81695cf9d4c1faf2b67580d44480e received from {username='slave'} at 127.16.36.193:42791
I20240502 08:05:37.385129 17387 tablet_copy_service.cc:433] P d8da072ac9514d3097eb441c8ee68428: ending tablet copy session d67b9894d6dc4495ad18c22ee4a0d458-38d81695cf9d4c1faf2b67580d44480e on tablet 38d81695cf9d4c1faf2b67580d44480e with peer d67b9894d6dc4495ad18c22ee4a0d458
I20240502 08:05:37.395362 19315 ts_tablet_manager.cc:902] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Initiating tablet copy from peer 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337)
I20240502 08:05:37.396073 19315 tablet_copy_client.cc:308] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:41337
I20240502 08:05:37.396667 17518 tablet_copy_service.cc:140] P 1256e2e5163d4240bd7dd791512ffa5c: Received BeginTabletCopySession request for tablet efbde964902249719b24cf743249ec6f from peer d8da072ac9514d3097eb441c8ee68428 ({username='slave'} at 127.16.36.194:45515)
I20240502 08:05:37.396790 17518 tablet_copy_service.cc:161] P 1256e2e5163d4240bd7dd791512ffa5c: Beginning new tablet copy session on tablet efbde964902249719b24cf743249ec6f from peer d8da072ac9514d3097eb441c8ee68428 at {username='slave'} at 127.16.36.194:45515: session id = d8da072ac9514d3097eb441c8ee68428-efbde964902249719b24cf743249ec6f
I20240502 08:05:37.397522 17518 tablet_copy_source_session.cc:205] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:37.398444 19315 ts_tablet_manager.cc:1875] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:37.400368 19315 ts_tablet_manager.cc:1888] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:37.400583 19315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efbde964902249719b24cf743249ec6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:37.401609 19315 tablet_copy_client.cc:783] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:37.401786 19315 tablet_copy_client.cc:647] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:37.403113 19315 tablet_copy_client.cc:523] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:37.404331 19315 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Bootstrap starting.
I20240502 08:05:37.424072 19315 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:37.424407 19315 tablet_bootstrap.cc:492] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Bootstrap complete.
I20240502 08:05:37.424597 19315 ts_tablet_manager.cc:1367] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Time spent bootstrapping tablet: real 0.020s user 0.021s sys 0.000s
I20240502 08:05:37.425005 19315 raft_consensus.cc:348] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } }
I20240502 08:05:37.425117 19315 raft_consensus.cc:729] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Initialized, Role: LEARNER
I20240502 08:05:37.425204 19315 consensus_queue.cc:260] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: true } }
I20240502 08:05:37.425443 19315 ts_tablet_manager.cc:1398] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:37.426132 17518 tablet_copy_service.cc:342] P 1256e2e5163d4240bd7dd791512ffa5c: Request end of tablet copy session d8da072ac9514d3097eb441c8ee68428-efbde964902249719b24cf743249ec6f received from {username='slave'} at 127.16.36.194:45515
I20240502 08:05:37.426218 17518 tablet_copy_service.cc:433] P 1256e2e5163d4240bd7dd791512ffa5c: ending tablet copy session d8da072ac9514d3097eb441c8ee68428-efbde964902249719b24cf743249ec6f on tablet efbde964902249719b24cf743249ec6f with peer d8da072ac9514d3097eb441c8ee68428
I20240502 08:05:37.695289 17234 raft_consensus.cc:1203] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:05:37.795744 17499 raft_consensus.cc:1203] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:05:37.880096 17234 raft_consensus.cc:1203] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 08:05:37.885573 17365 raft_consensus.cc:1203] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 08:05:38.216781 19364 raft_consensus.cc:1050] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d: attempting to promote NON_VOTER 1256e2e5163d4240bd7dd791512ffa5c to VOTER
I20240502 08:05:38.217223 19364 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } }
I20240502 08:05:38.218719 17499 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 LEARNER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:38.218816 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:38.218847 17234 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:38.219161 19364 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.219348 19363 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.219527 19388 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.221593 19388 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.222112 17499 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.223379 17365 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.224777 17234 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.226933 17088 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 8 to 9, 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "93ae1f0181ba4eae9801831b0e037f6d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.230998 18138 consensus_queue.cc:237] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } }
I20240502 08:05:38.232219 17499 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:38.232559 19363 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:38.233561 17365 raft_consensus.cc:1261] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Refusing update from remote peer 93ae1f0181ba4eae9801831b0e037f6d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:38.234062 19364 consensus_queue.cc:1035] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:38.234349 19363 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.235054 17365 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.236187 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df7c076e0edb4c9089c58c4a3d55eb83 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:38.236747 17091 catalog_manager.cc:5484] T df7c076e0edb4c9089c58c4a3d55eb83 P 93ae1f0181ba4eae9801831b0e037f6d reported cstate change: config changed from index 9 to 10, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "93ae1f0181ba4eae9801831b0e037f6d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:38.237061 17499 raft_consensus.cc:2932] T df7c076e0edb4c9089c58c4a3d55eb83 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "93ae1f0181ba4eae9801831b0e037f6d" member_type: VOTER last_known_addr { host: "127.16.36.197" port: 44867 } attrs { promote: false } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { promote: false } } }
I20240502 08:05:38.239570 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet df7c076e0edb4c9089c58c4a3d55eb83 with delete_type TABLET_DATA_TOMBSTONED (TS d67b9894d6dc4495ad18c22ee4a0d458 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33218
I20240502 08:05:38.240420 19406 tablet_replica.cc:330] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: stopping tablet replica
I20240502 08:05:38.241233 19406 raft_consensus.cc:2229] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:38.241487 19406 raft_consensus.cc:2258] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:38.242188 19406 ts_tablet_manager.cc:1875] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:38.244099 19406 ts_tablet_manager.cc:1888] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:05:38.244171 19406 log.cc:1196] T df7c076e0edb4c9089c58c4a3d55eb83 P d67b9894d6dc4495ad18c22ee4a0d458: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/df7c076e0edb4c9089c58c4a3d55eb83
I20240502 08:05:38.244740 17078 catalog_manager.cc:4830] TS d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): tablet df7c076e0edb4c9089c58c4a3d55eb83 (table rebalance_test_table_0 [id=d7d5b399b6514139aa634afe8ca22a7c]) successfully deleted
I20240502 08:05:38.314613 19365 raft_consensus.cc:1050] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c: attempting to promote NON_VOTER d8da072ac9514d3097eb441c8ee68428 to VOTER
I20240502 08:05:38.315022 19365 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } }
I20240502 08:05:38.316583 17365 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 LEARNER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:38.316655 17234 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:38.316655 18270 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:38.317121 19366 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:38.317314 19365 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:38.317345 18711 raft_consensus.cc:1050] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: attempting to promote NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 to VOTER
I20240502 08:05:38.317492 19433 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:38.317708 18711 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:38.319131 17234 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 LEARNER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:38.319201 18269 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:38.319243 17499 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:38.319636 19341 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:38.319936 19342 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:38.320199 18711 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:38.320336 19433 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.320885 17365 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.321061 18269 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.323613 17091 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 7 to 8, d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:38.323653 17235 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.324506 18711 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.324823 17499 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.326524 18270 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.327924 17084 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 7 to 8, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.328038 17499 consensus_queue.cc:237] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } }
I20240502 08:05:38.329605 18270 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:38.329936 17365 raft_consensus.cc:1261] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Refusing update from remote peer 1256e2e5163d4240bd7dd791512ffa5c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:38.329954 19366 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.330415 19366 consensus_queue.cc:1035] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.330533 17234 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.332049 19365 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.332549 18270 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.333905 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet efbde964902249719b24cf743249ec6f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:38.334023 17365 raft_consensus.cc:2932] T efbde964902249719b24cf743249ec6f P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } } }
I20240502 08:05:38.334503 17088 catalog_manager.cc:5484] T efbde964902249719b24cf743249ec6f P 1256e2e5163d4240bd7dd791512ffa5c reported cstate change: config changed from index 8 to 9, VOTER d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "1256e2e5163d4240bd7dd791512ffa5c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:38.337064 17214 tablet_service.cc:1514] Processing DeleteTablet for tablet efbde964902249719b24cf743249ec6f with delete_type TABLET_DATA_TOMBSTONED (TS d67b9894d6dc4495ad18c22ee4a0d458 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33218
I20240502 08:05:38.337297 19406 tablet_replica.cc:330] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: stopping tablet replica
I20240502 08:05:38.337450 19406 raft_consensus.cc:2229] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:38.337652 19406 raft_consensus.cc:2258] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:38.338606 19406 ts_tablet_manager.cc:1875] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:38.340508 19406 ts_tablet_manager.cc:1888] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 08:05:38.340580 19406 log.cc:1196] T efbde964902249719b24cf743249ec6f P d67b9894d6dc4495ad18c22ee4a0d458: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/efbde964902249719b24cf743249ec6f
I20240502 08:05:38.341042 17078 catalog_manager.cc:4830] TS d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425): tablet efbde964902249719b24cf743249ec6f (table rebalance_test_table_1 [id=6eef7a8221424c23aff9a4dffb449ea1]) successfully deleted
I20240502 08:05:38.341552 19436 raft_consensus.cc:1050] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428: attempting to promote NON_VOTER d67b9894d6dc4495ad18c22ee4a0d458 to VOTER
I20240502 08:05:38.341918 19436 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:38.343250 18270 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:38.343241 17234 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 LEARNER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:38.343276 17499 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:38.343575 18711 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.343748 19342 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.343928 19436 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:38.345968 18711 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.346463 17499 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.346571 17234 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.348613 17091 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 8 to 9, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:38.351714 18270 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } attrs { replace: true } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.351817 17365 consensus_queue.cc:237] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:38.353402 17234 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:38.354732 18269 raft_consensus.cc:1261] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Refusing update from remote peer d8da072ac9514d3097eb441c8ee68428: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:38.356734 19342 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:38.356930 19436 consensus_queue.cc:1035] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:38.358551 19341 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.359966 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38d81695cf9d4c1faf2b67580d44480e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:38.360505 18269 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.360541 17234 raft_consensus.cc:2932] T 38d81695cf9d4c1faf2b67580d44480e P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:38.360849 17084 catalog_manager.cc:5484] T 38d81695cf9d4c1faf2b67580d44480e P d8da072ac9514d3097eb441c8ee68428 reported cstate change: config changed from index 9 to 10, VOTER 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195) evicted. New cstate: current_term: 2 leader_uuid: "d8da072ac9514d3097eb441c8ee68428" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:38.364034 17477 tablet_service.cc:1514] Processing DeleteTablet for tablet 38d81695cf9d4c1faf2b67580d44480e with delete_type TABLET_DATA_TOMBSTONED (TS 1256e2e5163d4240bd7dd791512ffa5c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42336
I20240502 08:05:38.364528 19447 tablet_replica.cc:330] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: stopping tablet replica
I20240502 08:05:38.364723 19447 raft_consensus.cc:2229] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:38.365006 19447 raft_consensus.cc:2258] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:38.365754 19447 ts_tablet_manager.cc:1875] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:38.367542 19447 ts_tablet_manager.cc:1888] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:05:38.367620 19447 log.cc:1196] T 38d81695cf9d4c1faf2b67580d44480e P 1256e2e5163d4240bd7dd791512ffa5c: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/38d81695cf9d4c1faf2b67580d44480e
I20240502 08:05:38.368150 17078 catalog_manager.cc:4830] TS 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337): tablet 38d81695cf9d4c1faf2b67580d44480e (table rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]) successfully deleted
I20240502 08:05:38.492374 17365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63"
dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48532
I20240502 08:05:38.492591 17365 raft_consensus.cc:595] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:38.503216 17365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63"
dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48552
I20240502 08:05:38.503350 17365 raft_consensus.cc:595] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:38.714731 17365 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63"
dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48584
I20240502 08:05:38.714870 17365 raft_consensus.cc:595] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:38.731309 19341 raft_consensus.cc:979] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: : Instructing follower 7e6d75f768764435ab5f0b63159ca43a to start an election
I20240502 08:05:38.731429 19341 raft_consensus.cc:1067] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Signalling peer 7e6d75f768764435ab5f0b63159ca43a to start an election
I20240502 08:05:38.731930 18269 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63"
dest_uuid: "7e6d75f768764435ab5f0b63159ca43a"
from {username='slave'} at 127.16.36.194:39441
I20240502 08:05:38.732117 18269 raft_consensus.cc:482] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:38.732184 18269 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:38.733055 18269 raft_consensus.cc:504] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:38.734148 18269 leader_election.cc:288] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [CANDIDATE]: Term 2 election: Requested vote from peers d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347), 1256e2e5163d4240bd7dd791512ffa5c (127.16.36.195:41337), d67b9894d6dc4495ad18c22ee4a0d458 (127.16.36.193:46425)
I20240502 08:05:38.734390 17365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "7e6d75f768764435ab5f0b63159ca43a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d8da072ac9514d3097eb441c8ee68428"
I20240502 08:05:38.734514 17365 raft_consensus.cc:3032] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:38.734553 17365 raft_consensus.cc:729] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d8da072ac9514d3097eb441c8ee68428, State: Running, Role: LEADER
I20240502 08:05:38.734596 17234 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "7e6d75f768764435ab5f0b63159ca43a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d67b9894d6dc4495ad18c22ee4a0d458"
I20240502 08:05:38.734620 17499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0180a09549b4556addb20ce1f8d4e63" candidate_uuid: "7e6d75f768764435ab5f0b63159ca43a" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1256e2e5163d4240bd7dd791512ffa5c"
I20240502 08:05:38.734719 17234 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:38.734740 17499 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:38.734704 17365 consensus_queue.cc:260] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:38.734910 17365 raft_consensus.cc:3037] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:38.735528 17499 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e6d75f768764435ab5f0b63159ca43a in term 2.
I20240502 08:05:38.735805 17365 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e6d75f768764435ab5f0b63159ca43a in term 2.
I20240502 08:05:38.735805 17234 raft_consensus.cc:2454] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e6d75f768764435ab5f0b63159ca43a in term 2.
I20240502 08:05:38.736163 18204 leader_election.cc:302] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1256e2e5163d4240bd7dd791512ffa5c, 7e6d75f768764435ab5f0b63159ca43a, d8da072ac9514d3097eb441c8ee68428; no voters:
I20240502 08:05:38.736403 19441 raft_consensus.cc:2781] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:38.736649 19441 raft_consensus.cc:686] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 2 LEADER]: Becoming Leader. State: Replica: 7e6d75f768764435ab5f0b63159ca43a, State: Running, Role: LEADER
I20240502 08:05:38.736933 19441 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:38.739140 17084 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a reported cstate change: term changed from 1 to 2, leader changed from d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) to 7e6d75f768764435ab5f0b63159ca43a (127.16.36.196). New cstate: current_term: 2 leader_uuid: "7e6d75f768764435ab5f0b63159ca43a" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:39.125927 17234 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer 7e6d75f768764435ab5f0b63159ca43a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:39.126457 19441 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:39.172886 17499 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer 7e6d75f768764435ab5f0b63159ca43a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:39.173383 19441 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:39.176252 17365 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Refusing update from remote peer 7e6d75f768764435ab5f0b63159ca43a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 08:05:39.177403 19441 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8da072ac9514d3097eb441c8ee68428" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 35347 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:39.182497 18269 consensus_queue.cc:237] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } }
I20240502 08:05:39.183688 17234 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Refusing update from remote peer 7e6d75f768764435ab5f0b63159ca43a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:39.183737 17499 raft_consensus.cc:1261] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Refusing update from remote peer 7e6d75f768764435ab5f0b63159ca43a: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:39.184095 19488 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:39.184248 19502 consensus_queue.cc:1035] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:39.185874 19441 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:39.186340 17234 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P d67b9894d6dc4495ad18c22ee4a0d458 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:39.186333 17499 raft_consensus.cc:2932] T c0180a09549b4556addb20ce1f8d4e63 P 1256e2e5163d4240bd7dd791512ffa5c [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } } }
I20240502 08:05:39.188705 17078 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c0180a09549b4556addb20ce1f8d4e63 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:39.188714 17088 catalog_manager.cc:5484] T c0180a09549b4556addb20ce1f8d4e63 P 7e6d75f768764435ab5f0b63159ca43a reported cstate change: config changed from index 8 to 10, VOTER d8da072ac9514d3097eb441c8ee68428 (127.16.36.194) evicted. New cstate: current_term: 2 leader_uuid: "7e6d75f768764435ab5f0b63159ca43a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1256e2e5163d4240bd7dd791512ffa5c" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 41337 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7e6d75f768764435ab5f0b63159ca43a" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 38989 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d67b9894d6dc4495ad18c22ee4a0d458" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 46425 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:39.191394 17346 tablet_service.cc:1514] Processing DeleteTablet for tablet c0180a09549b4556addb20ce1f8d4e63 with delete_type TABLET_DATA_TOMBSTONED (TS d8da072ac9514d3097eb441c8ee68428 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58026
I20240502 08:05:39.191843 19507 tablet_replica.cc:330] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: stopping tablet replica
I20240502 08:05:39.191973 19507 raft_consensus.cc:2229] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:39.192093 19507 raft_consensus.cc:2258] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:39.192704 19507 ts_tablet_manager.cc:1875] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:39.194478 19507 ts_tablet_manager.cc:1888] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 08:05:39.194553 19507 log.cc:1196] T c0180a09549b4556addb20ce1f8d4e63 P d8da072ac9514d3097eb441c8ee68428: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.Rebalance_idx2_rf3_t3_l_3_2.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/c0180a09549b4556addb20ce1f8d4e63
I20240502 08:05:39.195119 17078 catalog_manager.cc:4830] TS d8da072ac9514d3097eb441c8ee68428 (127.16.36.194:35347): tablet c0180a09549b4556addb20ce1f8d4e63 (table rebalance_test_table_2 [id=7712d2dac5aa4281a239013e1acf7bdf]) successfully deleted
I20240502 08:05:39.391265 17477 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:39.392060 18250 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:39.392846 18119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:39.399976 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:39.408629 17214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:39.485553 17477 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:39.486086 18250 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:39.487927 17214 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:39.489552 17346 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 08:05:39.496836 18119 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 08:05:39.535832 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17153
I20240502 08:05:39.543063 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17284
I20240502 08:05:39.549656 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17416
I20240502 08:05:39.556404 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 18187
I20240502 08:05:39.562433 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 18057
I20240502 08:05:39.568499 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 17062
2024-05-02T08:05:39Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx2_rf3_t3_l_3_2 (15873 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1
I20240502 08:05:39.596911 16531 test_util.cc:268] Using random seed: 295687785
I20240502 08:05:39.597733 16531 ts_itest-base.cc:112] Starting cluster with:
I20240502 08:05:39.597775 16531 ts_itest-base.cc:113] --------------
I20240502 08:05:39.597786 16531 ts_itest-base.cc:114] 4 tablet servers
I20240502 08:05:39.597797 16531 ts_itest-base.cc:115] 3 replicas per TS
I20240502 08:05:39.597807 16531 ts_itest-base.cc:116] --------------
2024-05-02T08:05:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T08:05:39Z Disabled control of system clock
I20240502 08:05:39.606011 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:32833
--webserver_interface=127.16.36.254
--webserver_port=0
--builtin_ntp_servers=127.16.36.212:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.36.254:32833
--location_mapping_cmd=/tmp/dist-test-taskm5AjVx/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 08:05:39.711751 19579 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 08:05:39.712044 19579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:39.712102 19579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:39.712177 19579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:39.715304 19579 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:39.715410 19579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 08:05:39.715464 19579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:39.715503 19579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 08:05:39.715543 19579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 08:05:39.719275 19579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskm5AjVx/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:1
--ipki_ca_key_size=768
--master_addresses=127.16.36.254:32833
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.36.254:32833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.16.36.254
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.19579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:39.720232 19579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:39.721426 19579 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:39.728727 19584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:39.729147 19585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:39.729982 19587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:39.730726 19579 server_base.cc:1034] running on GCE node
I20240502 08:05:39.731245 19579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:39.732204 19579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:39.733361 19579 hybrid_clock.cc:648] HybridClock initialized: now 1714637139733345 us; error 30 us; skew 500 ppm
I20240502 08:05:39.735285 19579 webserver.cc:460] Webserver started at http://127.16.36.254:37893/ using document root <none> and password file <none>
I20240502 08:05:39.735846 19579 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:39.735910 19579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:39.736079 19579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:39.737748 19579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1"
format_stamp: "Formatted at 2024-05-02 08:05:39 on dist-test-slave-9sjq"
I20240502 08:05:39.738219 19579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1"
format_stamp: "Formatted at 2024-05-02 08:05:39 on dist-test-slave-9sjq"
I20240502 08:05:39.741461 19579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 08:05:39.743584 19593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:39.744601 19579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:39.744719 19579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1"
format_stamp: "Formatted at 2024-05-02 08:05:39 on dist-test-slave-9sjq"
I20240502 08:05:39.744815 19579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:39.757129 19579 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:39.757797 19579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:39.757961 19579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:39.764451 19579 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.254:32833
I20240502 08:05:39.764482 19645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.254:32833 every 8 connection(s)
I20240502 08:05:39.765488 19579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 08:05:39.768078 19646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:39.771152 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 19579
I20240502 08:05:39.771310 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 08:05:39.773614 19646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1: Bootstrap starting.
I20240502 08:05:39.775713 19646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:39.776403 19646 log.cc:824] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:39.778331 19646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1: No bootstrap required, opened a new log
I20240502 08:05:39.781298 19646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 32833 } }
I20240502 08:05:39.781482 19646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:39.781515 19646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d7dedb011e4aa78a08c31aa7fbd7e1, State: Initialized, Role: FOLLOWER
I20240502 08:05:39.781864 19646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 32833 } }
I20240502 08:05:39.781965 19646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 08:05:39.782009 19646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 08:05:39.782094 19646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:39.782811 19646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 32833 } }
I20240502 08:05:39.783146 19646 leader_election.cc:302] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54d7dedb011e4aa78a08c31aa7fbd7e1; no voters:
I20240502 08:05:39.783419 19646 leader_election.cc:288] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 08:05:39.783555 19651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:39.783783 19651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [term 1 LEADER]: Becoming Leader. State: Replica: 54d7dedb011e4aa78a08c31aa7fbd7e1, State: Running, Role: LEADER
I20240502 08:05:39.784066 19651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 32833 } }
I20240502 08:05:39.784482 19646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 08:05:39.785599 19653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54d7dedb011e4aa78a08c31aa7fbd7e1. Latest consensus state: current_term: 1 leader_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 32833 } } }
I20240502 08:05:39.785750 19653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:39.785989 19652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d7dedb011e4aa78a08c31aa7fbd7e1" member_type: VOTER last_known_addr { host: "127.16.36.254" port: 32833 } } }
I20240502 08:05:39.786082 19652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1 [sys.catalog]: This master's current role is: LEADER
I20240502 08:05:39.786167 19660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 08:05:39.789891 19660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 08:05:39.794859 19660 catalog_manager.cc:1347] Generated new cluster ID: 4fd0946c02be4adb993fe2b3888cdd77
I20240502 08:05:39.794937 19660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 08:05:39.817950 19660 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 08:05:39.819280 19660 catalog_manager.cc:1504] Loading token signing keys...
I20240502 08:05:39.828977 19660 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 54d7dedb011e4aa78a08c31aa7fbd7e1: Generated new TSK 0
I20240502 08:05:39.829842 19660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 08:05:39.832720 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.193:0
--local_ip_for_outbound_sockets=127.16.36.193
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--builtin_ntp_servers=127.16.36.212:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:05:39.942440 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:39.942679 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:39.942761 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:39.945967 19670 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:39.946062 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:39.946164 19670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.193
I20240502 08:05:39.950404 19670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.16.36.193
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--never_fsync=true
--heap_profile_path=/tmp/kudu.19670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.193
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:39.951368 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:39.952617 19670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:39.961478 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:39.961805 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:39.962388 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:39.963284 19670 server_base.cc:1034] running on GCE node
I20240502 08:05:39.963735 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:39.964340 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:39.965512 19670 hybrid_clock.cc:648] HybridClock initialized: now 1714637139965498 us; error 31 us; skew 500 ppm
I20240502 08:05:39.967608 19670 webserver.cc:460] Webserver started at http://127.16.36.193:43017/ using document root <none> and password file <none>
I20240502 08:05:39.968169 19670 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:39.968230 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:39.968407 19670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:39.970109 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
format_stamp: "Formatted at 2024-05-02 08:05:39 on dist-test-slave-9sjq"
I20240502 08:05:39.970582 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
format_stamp: "Formatted at 2024-05-02 08:05:39 on dist-test-slave-9sjq"
I20240502 08:05:39.974066 19670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20240502 08:05:39.976362 19684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:39.977473 19670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:39.977578 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
format_stamp: "Formatted at 2024-05-02 08:05:39 on dist-test-slave-9sjq"
I20240502 08:05:39.977664 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:39.993755 19670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:39.994407 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:39.994573 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:39.995111 19670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:39.995999 19670 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:39.996049 19670 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:39.996109 19670 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:39.996136 19670 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.004818 19670 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.193:36001
I20240502 08:05:40.004895 19797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.193:36001 every 8 connection(s)
I20240502 08:05:40.006211 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 08:05:40.008994 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 19670
I20240502 08:05:40.009305 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 08:05:40.012871 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.194:0
--local_ip_for_outbound_sockets=127.16.36.194
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--builtin_ntp_servers=127.16.36.212:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:40.018935 19798 heartbeater.cc:347] Connected to a master server at 127.16.36.254:32833
I20240502 08:05:40.019300 19798 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:40.020043 19798 heartbeater.cc:511] Master 127.16.36.254:32833 requested a full tablet report, sending...
I20240502 08:05:40.059988 19610 ts_manager.cc:194] Registered new tserver with Master: 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:40.061383 19610 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.193:58565
W20240502 08:05:40.126595 19801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:40.126817 19801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:40.126901 19801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:40.129930 19801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:40.130002 19801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:40.130105 19801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.194
I20240502 08:05:40.133667 19801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.16.36.194
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--never_fsync=true
--heap_profile_path=/tmp/kudu.19801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.194
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:40.134577 19801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:40.135668 19801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:40.144309 19810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:40.144667 19808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:40.145061 19807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:40.146049 19801 server_base.cc:1034] running on GCE node
I20240502 08:05:40.146466 19801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:40.147080 19801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:40.148259 19801 hybrid_clock.cc:648] HybridClock initialized: now 1714637140148234 us; error 42 us; skew 500 ppm
I20240502 08:05:40.150316 19801 webserver.cc:460] Webserver started at http://127.16.36.194:37959/ using document root <none> and password file <none>
I20240502 08:05:40.150879 19801 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:40.150945 19801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:40.151127 19801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:40.152889 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.153378 19801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.156811 19801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 08:05:40.159050 19816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.160092 19801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:40.160219 19801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.160316 19801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:40.179368 19801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:40.180089 19801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:40.180279 19801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:40.180943 19801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:40.181921 19801 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:40.181979 19801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.182039 19801 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:40.182075 19801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.191112 19801 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.194:45299
I20240502 08:05:40.191143 19929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.194:45299 every 8 connection(s)
I20240502 08:05:40.192101 19801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 08:05:40.199329 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 19801
I20240502 08:05:40.199458 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 08:05:40.201643 19930 heartbeater.cc:347] Connected to a master server at 127.16.36.254:32833
I20240502 08:05:40.201941 19930 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:40.201974 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.195:0
--local_ip_for_outbound_sockets=127.16.36.195
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--builtin_ntp_servers=127.16.36.212:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:40.202495 19930 heartbeater.cc:511] Master 127.16.36.254:32833 requested a full tablet report, sending...
I20240502 08:05:40.239455 19609 ts_manager.cc:194] Registered new tserver with Master: a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:40.240428 19609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.194:38037
W20240502 08:05:40.315649 19933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:40.315985 19933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:40.316030 19933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:40.319203 19933 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:40.319278 19933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:40.319346 19933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.195
I20240502 08:05:40.322978 19933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.16.36.195
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--never_fsync=true
--heap_profile_path=/tmp/kudu.19933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.195
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:40.323871 19933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:40.325019 19933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:40.333355 19939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:40.333930 19940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:40.334559 19942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:40.335372 19933 server_base.cc:1034] running on GCE node
I20240502 08:05:40.335763 19933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:40.336333 19933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:40.337488 19933 hybrid_clock.cc:648] HybridClock initialized: now 1714637140337475 us; error 34 us; skew 500 ppm
I20240502 08:05:40.339444 19933 webserver.cc:460] Webserver started at http://127.16.36.195:45339/ using document root <none> and password file <none>
I20240502 08:05:40.339999 19933 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:40.340062 19933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:40.340235 19933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:40.341922 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.342396 19933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.345678 19933 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20240502 08:05:40.347921 19948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.349001 19933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 08:05:40.349117 19933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.349234 19933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:40.363292 19933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:40.363961 19933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:40.364123 19933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:40.364714 19933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:40.365604 19933 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:40.365651 19933 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.365697 19933 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:40.365727 19933 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.373926 19933 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.195:44147
I20240502 08:05:40.373980 20061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.195:44147 every 8 connection(s)
I20240502 08:05:40.374856 19933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 08:05:40.378515 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 19933
I20240502 08:05:40.378651 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 08:05:40.380846 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:0
--local_ip_for_outbound_sockets=127.16.36.196
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--builtin_ntp_servers=127.16.36.212:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 08:05:40.383942 20062 heartbeater.cc:347] Connected to a master server at 127.16.36.254:32833
I20240502 08:05:40.384166 20062 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:40.384765 20062 heartbeater.cc:511] Master 127.16.36.254:32833 requested a full tablet report, sending...
I20240502 08:05:40.423557 19609 ts_manager.cc:194] Registered new tserver with Master: 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:40.424268 19609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.195:39077
W20240502 08:05:40.489338 20065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:40.489570 20065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:40.489646 20065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:40.492679 20065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:40.492753 20065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:40.492846 20065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 08:05:40.496435 20065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=0
--tserver_master_addrs=127.16.36.254:32833
--never_fsync=true
--heap_profile_path=/tmp/kudu.20065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:40.497416 20065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:40.498538 20065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:40.507045 20074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:40.507438 20071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:40.508280 20072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:40.509111 20065 server_base.cc:1034] running on GCE node
I20240502 08:05:40.509560 20065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:40.510135 20065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:40.511317 20065 hybrid_clock.cc:648] HybridClock initialized: now 1714637140511291 us; error 41 us; skew 500 ppm
I20240502 08:05:40.513429 20065 webserver.cc:460] Webserver started at http://127.16.36.196:36455/ using document root <none> and password file <none>
I20240502 08:05:40.514009 20065 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:40.514072 20065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:40.514250 20065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 08:05:40.515939 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cc250aaf098d4d10b71f6aa076a47386"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.516417 20065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cc250aaf098d4d10b71f6aa076a47386"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.519876 20065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240502 08:05:40.522032 20080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.522985 20065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 08:05:40.523093 20065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cc250aaf098d4d10b71f6aa076a47386"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:40.523177 20065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:40.535298 20065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:40.535866 20065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:40.536012 20065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:40.536533 20065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:40.537407 20065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:40.537457 20065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.537518 20065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:40.537544 20065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:40.545928 20065 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:35635
I20240502 08:05:40.545945 20193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:35635 every 8 connection(s)
I20240502 08:05:40.546847 20065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:05:40.547138 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 20065
I20240502 08:05:40.547253 16531 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 08:05:40.555531 20194 heartbeater.cc:347] Connected to a master server at 127.16.36.254:32833
I20240502 08:05:40.555734 20194 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:40.556228 20194 heartbeater.cc:511] Master 127.16.36.254:32833 requested a full tablet report, sending...
I20240502 08:05:40.590703 19609 ts_manager.cc:194] Registered new tserver with Master: cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635)
I20240502 08:05:40.591420 19609 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:44721
I20240502 08:05:40.598029 16531 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 08:05:40.606835 16531 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 08:05:40.606914 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 20065
I20240502 08:05:41.063915 19798 heartbeater.cc:503] Master 127.16.36.254:32833 was elected leader, sending a full tablet report...
I20240502 08:05:41.242815 19930 heartbeater.cc:503] Master 127.16.36.254:32833 was elected leader, sending a full tablet report...
I20240502 08:05:41.426565 20062 heartbeater.cc:503] Master 127.16.36.254:32833 was elected leader, sending a full tablet report...
I20240502 08:05:44.364748 19609 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40854:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 08:05:44.368109 19609 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:44.386476 19732 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8d28c74e4b4be089a0d9cea5331e3c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.386473 19862 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf541bb2ff945a386808e1b025cfec0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.386476 19864 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8d28c74e4b4be089a0d9cea5331e3c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.387605 19864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d28c74e4b4be089a0d9cea5331e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.387607 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d28c74e4b4be089a0d9cea5331e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.387862 19730 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf541bb2ff945a386808e1b025cfec0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.388136 19730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf541bb2ff945a386808e1b025cfec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.389037 19731 tablet_service.cc:1467] Processing CreateTablet for tablet f9685f0c65734d7bb8bea0afe792cdd7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.389284 19731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9685f0c65734d7bb8bea0afe792cdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.389878 19863 tablet_service.cc:1467] Processing CreateTablet for tablet f9685f0c65734d7bb8bea0afe792cdd7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.390156 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9685f0c65734d7bb8bea0afe792cdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.391348 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf541bb2ff945a386808e1b025cfec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.395220 20214 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:44.396207 20213 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:44.397442 20214 tablet_bootstrap.cc:654] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.398171 20214 log.cc:824] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:44.398851 20213 tablet_bootstrap.cc:654] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.399039 19995 tablet_service.cc:1467] Processing CreateTablet for tablet f9685f0c65734d7bb8bea0afe792cdd7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.399039 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 3f8d28c74e4b4be089a0d9cea5331e3c (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.399649 20213 log.cc:824] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:44.399350 19994 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf541bb2ff945a386808e1b025cfec0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.400168 19995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9685f0c65734d7bb8bea0afe792cdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.401350 20214 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:44.401425 20213 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:44.401546 20214 ts_tablet_manager.cc:1367] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 08:05:44.401604 20213 ts_tablet_manager.cc:1367] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 08:05:44.404171 20214 raft_consensus.cc:348] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.404362 20214 raft_consensus.cc:374] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.404422 20214 raft_consensus.cc:729] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.404878 20214 consensus_queue.cc:260] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.405690 20213 raft_consensus.cc:348] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.405833 20214 ts_tablet_manager.cc:1398] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 08:05:44.405900 20213 raft_consensus.cc:374] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.405961 20213 raft_consensus.cc:729] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.406219 20214 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:44.406387 20213 consensus_queue.cc:260] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.407362 20213 ts_tablet_manager.cc:1398] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 08:05:44.407711 20213 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:44.408218 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf541bb2ff945a386808e1b025cfec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.408766 20213 tablet_bootstrap.cc:654] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.409508 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d28c74e4b4be089a0d9cea5331e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.411747 20214 tablet_bootstrap.cc:654] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.413026 20219 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:44.413071 20214 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:44.413110 20213 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:44.413157 20214 ts_tablet_manager.cc:1367] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 08:05:44.413199 20213 ts_tablet_manager.cc:1367] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.002s
I20240502 08:05:44.413694 20214 raft_consensus.cc:348] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.413813 20214 raft_consensus.cc:374] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.413756 20213 raft_consensus.cc:348] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.413851 20214 raft_consensus.cc:729] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.413873 20213 raft_consensus.cc:374] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.413906 20213 raft_consensus.cc:729] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.413985 20214 consensus_queue.cc:260] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.414048 20213 consensus_queue.cc:260] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.414263 20214 ts_tablet_manager.cc:1398] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.414314 20213 ts_tablet_manager.cc:1398] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:44.414422 20214 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:44.414467 20213 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:44.415325 20213 tablet_bootstrap.cc:654] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.415339 20214 tablet_bootstrap.cc:654] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.415418 20219 tablet_bootstrap.cc:654] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.416268 20219 log.cc:824] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:44.416656 20214 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:44.416743 20214 ts_tablet_manager.cc:1367] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:44.417263 20214 raft_consensus.cc:348] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.417397 20214 raft_consensus.cc:374] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.417446 20214 raft_consensus.cc:729] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.417593 20214 consensus_queue.cc:260] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.417884 20214 ts_tablet_manager.cc:1398] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:44.418598 20213 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:44.418687 20213 ts_tablet_manager.cc:1367] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 08:05:44.419145 20213 raft_consensus.cc:348] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.419251 20213 raft_consensus.cc:374] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.419281 20213 raft_consensus.cc:729] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.419380 20219 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:44.419400 20213 consensus_queue.cc:260] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.419590 20219 ts_tablet_manager.cc:1367] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 08:05:44.419690 20213 ts_tablet_manager.cc:1398] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:44.422646 20219 raft_consensus.cc:348] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.422863 20219 raft_consensus.cc:374] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.422935 20219 raft_consensus.cc:729] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.423337 20219 consensus_queue.cc:260] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.424249 20219 ts_tablet_manager.cc:1398] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 08:05:44.424718 20219 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:44.425838 20219 tablet_bootstrap.cc:654] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.427157 20219 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:44.427271 20219 ts_tablet_manager.cc:1367] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:05:44.427853 20219 raft_consensus.cc:348] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.427955 20219 raft_consensus.cc:374] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.427978 20219 raft_consensus.cc:729] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.428086 20219 consensus_queue.cc:260] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.428321 20219 ts_tablet_manager.cc:1398] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:44.428521 20219 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:44.429502 20219 tablet_bootstrap.cc:654] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.430668 20219 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:44.430761 20219 ts_tablet_manager.cc:1367] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:44.431313 20219 raft_consensus.cc:348] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.431414 20219 raft_consensus.cc:374] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.431437 20219 raft_consensus.cc:729] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.431566 20219 consensus_queue.cc:260] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.431834 20219 ts_tablet_manager.cc:1398] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240502 08:05:44.446213 19931 tablet.cc:2369] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:05:44.482026 20225 raft_consensus.cc:482] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:44.482282 20225 raft_consensus.cc:504] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.483457 20225 leader_election.cc:288] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:44.488744 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9685f0c65734d7bb8bea0afe792cdd7" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" is_pre_election: true
I20240502 08:05:44.489107 19752 raft_consensus.cc:2454] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 0.
I20240502 08:05:44.489202 19884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9685f0c65734d7bb8bea0afe792cdd7" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:44.489512 19884 raft_consensus.cc:2454] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 0.
I20240502 08:05:44.489542 19951 leader_election.cc:302] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, 9e4a71b756c544e4ae6c8cbe92b6eb5b; no voters:
I20240502 08:05:44.489848 20225 raft_consensus.cc:2781] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:44.489939 20225 raft_consensus.cc:482] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:44.489988 20225 raft_consensus.cc:3037] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.490906 20225 raft_consensus.cc:504] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.491269 20225 leader_election.cc:288] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:44.491585 19884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9685f0c65734d7bb8bea0afe792cdd7" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:44.491629 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9685f0c65734d7bb8bea0afe792cdd7" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:44.491690 19884 raft_consensus.cc:3037] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.491776 19752 raft_consensus.cc:3037] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.492642 19884 raft_consensus.cc:2454] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 1.
I20240502 08:05:44.492960 19949 leader_election.cc:302] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:44.493157 20225 raft_consensus.cc:2781] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:44.493157 19752 raft_consensus.cc:2454] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 1.
I20240502 08:05:44.493388 20225 raft_consensus.cc:686] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:44.493757 20225 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.496886 19609 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } }
W20240502 08:05:44.511132 19799 tablet.cc:2369] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 08:05:44.543390 20217 raft_consensus.cc:482] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:44.543545 20217 raft_consensus.cc:504] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.544705 20217 leader_election.cc:288] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:44.548429 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d28c74e4b4be089a0d9cea5331e3c" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" is_pre_election: true
I20240502 08:05:44.548529 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d28c74e4b4be089a0d9cea5331e3c" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:44.548645 19752 raft_consensus.cc:2454] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 0.
I20240502 08:05:44.548789 20016 raft_consensus.cc:2454] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 0.
I20240502 08:05:44.548993 19819 leader_election.cc:302] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:44.549244 20217 raft_consensus.cc:2781] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:44.549309 20217 raft_consensus.cc:482] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:44.549345 20217 raft_consensus.cc:3037] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.550172 20217 raft_consensus.cc:504] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.550521 20217 leader_election.cc:288] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:44.550882 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d28c74e4b4be089a0d9cea5331e3c" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:44.550882 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f8d28c74e4b4be089a0d9cea5331e3c" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:44.551012 20016 raft_consensus.cc:3037] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.551012 19752 raft_consensus.cc:3037] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.552009 19752 raft_consensus.cc:2454] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 1.
I20240502 08:05:44.552009 20016 raft_consensus.cc:2454] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 1.
I20240502 08:05:44.552335 19820 leader_election.cc:302] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:44.552517 20217 raft_consensus.cc:2781] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:44.552754 20217 raft_consensus.cc:686] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Becoming Leader. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Running, Role: LEADER
I20240502 08:05:44.553081 20217 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.556151 19609 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:44.611236 20218 raft_consensus.cc:482] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:44.611419 20218 raft_consensus.cc:504] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.612565 20218 leader_election.cc:288] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:44.616361 19884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:44.616361 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:44.616560 20016 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:44.616560 19884 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:44.616959 19685 leader_election.cc:302] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:44.617269 20218 raft_consensus.cc:2781] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:44.617336 20218 raft_consensus.cc:482] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:44.617383 20218 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.618371 20218 raft_consensus.cc:504] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.618779 20218 leader_election.cc:288] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:44.619154 19884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:44.619304 19884 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.619266 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:44.619385 20016 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.620227 20016 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:44.620225 19884 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:44.620546 19685 leader_election.cc:302] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:44.620728 20218 raft_consensus.cc:2781] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:44.620988 20218 raft_consensus.cc:686] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Leader. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:44.621345 20218 consensus_queue.cc:237] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.624320 19609 catalog_manager.cc:5484] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:44.629261 20063 tablet.cc:2369] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 08:05:44.786278 20240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:44.786585 20240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:44.789655 20240 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 08:05:44.891835 20016 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:44.892515 20217 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:44.893815 19752 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:44.893815 19884 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:44.894284 19751 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:44.894492 20225 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:44.894697 20230 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:44.896303 20015 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:44.896566 20234 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:44.896889 20218 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:44.901011 19883 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:44.902940 20218 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:44.911166 20272 mvcc.cc:205] Tried to move back new op lower bound from 7023153745541984256 to 7023153744092196864. Current Snapshot: MvccSnapshot[applied={T|T < 7023153745468837888}]
I20240502 08:05:44.911427 20274 mvcc.cc:205] Tried to move back new op lower bound from 7023153745476112384 to 7023153743849222144. Current Snapshot: MvccSnapshot[applied={T|T < 7023153745476112384}]
I20240502 08:05:44.917546 20273 mvcc.cc:205] Tried to move back new op lower bound from 7023153745542533120 to 7023153743849222144. Current Snapshot: MvccSnapshot[applied={T|T < 7023153745476112384}]
INSERT report
rows total: 2000
time total: 126.214 ms
time per row: 0.0631072 ms
I20240502 08:05:44.964102 19605 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40854:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 08:05:44.964705 19605 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:44.970233 19862 tablet_service.cc:1467] Processing CreateTablet for tablet 12a28c958e9d4493a112424486c56534 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.970235 19732 tablet_service.cc:1467] Processing CreateTablet for tablet 12a28c958e9d4493a112424486c56534 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.970551 19862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a28c958e9d4493a112424486c56534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.970562 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a28c958e9d4493a112424486c56534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.970840 19994 tablet_service.cc:1467] Processing CreateTablet for tablet da90061bf3d9471896a2e51bb02864bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.971102 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90061bf3d9471896a2e51bb02864bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.972810 19995 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee8e3ede5eb4640bbb530277175dd52 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.973063 19995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee8e3ede5eb4640bbb530277175dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.973511 20300 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:44.974042 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 12a28c958e9d4493a112424486c56534 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.974040 19732 tablet_service.cc:1467] Processing CreateTablet for tablet da90061bf3d9471896a2e51bb02864bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.974277 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90061bf3d9471896a2e51bb02864bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.974277 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a28c958e9d4493a112424486c56534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.974853 19730 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee8e3ede5eb4640bbb530277175dd52 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.975073 19730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee8e3ede5eb4640bbb530277175dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.976729 19863 tablet_service.cc:1467] Processing CreateTablet for tablet da90061bf3d9471896a2e51bb02864bb (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.976912 20300 tablet_bootstrap.cc:654] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.977262 20299 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:44.977231 19864 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee8e3ede5eb4640bbb530277175dd52 (DEFAULT_TABLE table=rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:44.977567 19864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee8e3ede5eb4640bbb530277175dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.977773 20301 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:44.978211 20299 tablet_bootstrap.cc:654] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.978858 20301 tablet_bootstrap.cc:654] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.979338 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90061bf3d9471896a2e51bb02864bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:44.979753 20299 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:44.979838 20299 ts_tablet_manager.cc:1367] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:44.980389 20300 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:44.980372 20299 raft_consensus.cc:348] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.980506 20300 ts_tablet_manager.cc:1367] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240502 08:05:44.980522 20299 raft_consensus.cc:374] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.980558 20299 raft_consensus.cc:729] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.980615 20301 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:44.980686 20301 ts_tablet_manager.cc:1367] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 08:05:44.980700 20299 consensus_queue.cc:260] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.981024 20299 ts_tablet_manager.cc:1398] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.980983 20300 raft_consensus.cc:348] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.981099 20300 raft_consensus.cc:374] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.981137 20300 raft_consensus.cc:729] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.981199 20299 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:44.981134 20301 raft_consensus.cc:348] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.981256 20301 raft_consensus.cc:374] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.981295 20301 raft_consensus.cc:729] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.981257 20300 consensus_queue.cc:260] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.981413 20301 consensus_queue.cc:260] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.981549 20300 ts_tablet_manager.cc:1398] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.981681 20301 ts_tablet_manager.cc:1398] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.981714 20300 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:44.981860 20301 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:44.982609 20300 tablet_bootstrap.cc:654] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.982772 20301 tablet_bootstrap.cc:654] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.982906 20299 tablet_bootstrap.cc:654] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.984259 20299 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:44.984352 20299 ts_tablet_manager.cc:1367] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:44.984846 20299 raft_consensus.cc:348] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.984946 20299 raft_consensus.cc:374] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.984953 20301 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:44.984982 20299 raft_consensus.cc:729] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.985026 20301 ts_tablet_manager.cc:1367] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:05:44.985106 20299 consensus_queue.cc:260] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.985364 20299 ts_tablet_manager.cc:1398] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.985558 20299 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:44.985617 20301 raft_consensus.cc:348] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.985703 20301 raft_consensus.cc:374] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.985733 20301 raft_consensus.cc:729] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.985869 20301 consensus_queue.cc:260] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.986263 20301 ts_tablet_manager.cc:1398] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.986416 20301 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:44.986502 20299 tablet_bootstrap.cc:654] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.986627 20300 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:44.986711 20300 ts_tablet_manager.cc:1367] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 08:05:44.987165 20300 raft_consensus.cc:348] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.987282 20300 raft_consensus.cc:374] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.987318 20300 raft_consensus.cc:729] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.987358 20301 tablet_bootstrap.cc:654] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.987644 20300 consensus_queue.cc:260] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.987741 20299 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:44.987818 20299 ts_tablet_manager.cc:1367] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:44.988283 20299 raft_consensus.cc:348] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.988377 20299 raft_consensus.cc:374] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.988407 20299 raft_consensus.cc:729] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.988624 20301 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:44.988694 20301 ts_tablet_manager.cc:1367] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:44.988672 20299 consensus_queue.cc:260] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.988947 20299 ts_tablet_manager.cc:1398] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:44.989102 20301 raft_consensus.cc:348] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.989207 20301 raft_consensus.cc:374] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.989248 20301 raft_consensus.cc:729] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.989368 20300 ts_tablet_manager.cc:1398] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 08:05:44.989362 20301 consensus_queue.cc:260] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.989545 20300 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:44.989606 20301 ts_tablet_manager.cc:1398] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:44.990530 20300 tablet_bootstrap.cc:654] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:44.991849 20300 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:44.991945 20300 ts_tablet_manager.cc:1367] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240502 08:05:44.992328 20300 raft_consensus.cc:348] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.992415 20300 raft_consensus.cc:374] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:44.992448 20300 raft_consensus.cc:729] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:44.992591 20300 consensus_queue.cc:260] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.992834 20300 ts_tablet_manager.cc:1398] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:44.994935 20218 raft_consensus.cc:482] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:44.995035 20218 raft_consensus.cc:504] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.995438 20218 leader_election.cc:288] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:44.995807 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:44.995857 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:44.995961 19883 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:44.995975 20015 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:44.996237 19688 leader_election.cc:302] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, 9e4a71b756c544e4ae6c8cbe92b6eb5b; no voters:
I20240502 08:05:44.996492 20218 raft_consensus.cc:2781] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:44.996548 20218 raft_consensus.cc:482] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:44.996595 20218 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.997470 20218 raft_consensus.cc:504] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:44.997807 20218 leader_election.cc:288] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:44.998152 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:44.998273 19883 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.998293 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:44.998458 20015 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:44.999217 19883 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:44.999294 20015 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:44.999544 19685 leader_election.cc:302] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:44.999724 20218 raft_consensus.cc:2781] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:44.999823 20218 raft_consensus.cc:686] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Leader. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:44.999984 20218 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.001691 19605 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:45.185675 20218 raft_consensus.cc:482] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:45.185926 20218 raft_consensus.cc:504] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.186513 20218 leader_election.cc:288] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.187043 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:45.187044 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:45.187180 19883 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:45.187220 20015 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:45.187520 19685 leader_election.cc:302] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.187747 20218 raft_consensus.cc:2781] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:45.187808 20218 raft_consensus.cc:482] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:45.187837 20218 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.188740 20218 raft_consensus.cc:504] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.189110 20218 leader_election.cc:288] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.189512 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:45.189654 20015 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.190443 20015 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:45.190711 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:45.190771 19688 leader_election.cc:302] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, 9e4a71b756c544e4ae6c8cbe92b6eb5b; no voters:
I20240502 08:05:45.190861 19883 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.191004 20218 raft_consensus.cc:2781] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:45.191116 20218 raft_consensus.cc:686] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Leader. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:45.191293 20218 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.192965 19883 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:45.193830 19605 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:45.315975 20218 raft_consensus.cc:482] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:45.316110 20218 raft_consensus.cc:504] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.316530 20218 leader_election.cc:288] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.316887 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:45.317065 19883 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:45.317029 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:45.317162 20015 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:45.317390 19685 leader_election.cc:302] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.317585 20218 raft_consensus.cc:2781] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:45.317669 20218 raft_consensus.cc:482] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:45.317704 20218 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.318476 20218 raft_consensus.cc:504] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.318850 20218 leader_election.cc:288] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.319269 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:45.319253 20015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:45.319386 20015 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.319389 19883 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.320310 19883 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:45.320310 20015 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:45.320626 19685 leader_election.cc:302] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.320806 20218 raft_consensus.cc:2781] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:45.320884 20218 raft_consensus.cc:686] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Leader. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:45.321048 20218 consensus_queue.cc:237] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.322599 19605 catalog_manager.cc:5484] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:45.452069 20314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:45.452351 20314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:45.455710 20314 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 08:05:45.533180 20015 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.534449 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.534508 19882 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.534466 19884 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.534770 20015 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.534770 20016 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.535849 20298 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.536183 20298 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.536885 20218 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.537144 20218 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.537411 20218 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.537778 20298 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.917 ms
time per row: 0.0519586 ms
I20240502 08:05:45.601768 19601 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40854:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 08:05:45.602284 19601 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 08:05:45.607373 19730 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4fb281e443457fac00ceaa3e1df876 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.607590 19732 tablet_service.cc:1467] Processing CreateTablet for tablet b44e8f73540a40d080c94498e12b1148 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.607705 19730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fb281e443457fac00ceaa3e1df876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.607875 19732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e8f73540a40d080c94498e12b1148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.608402 19996 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4fb281e443457fac00ceaa3e1df876 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.608489 19995 tablet_service.cc:1467] Processing CreateTablet for tablet b44e8f73540a40d080c94498e12b1148 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.608692 19996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fb281e443457fac00ceaa3e1df876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.608903 19994 tablet_service.cc:1467] Processing CreateTablet for tablet 179b4d44bff84b84b74d66c98a506f50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.609117 19994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179b4d44bff84b84b74d66c98a506f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.609966 19731 tablet_service.cc:1467] Processing CreateTablet for tablet 179b4d44bff84b84b74d66c98a506f50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.610163 19731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179b4d44bff84b84b74d66c98a506f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.610311 19863 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4fb281e443457fac00ceaa3e1df876 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.610574 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fb281e443457fac00ceaa3e1df876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.611126 20373 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:45.611505 19995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e8f73540a40d080c94498e12b1148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.612035 20373 tablet_bootstrap.cc:654] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.613178 20373 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:45.613255 20373 ts_tablet_manager.cc:1367] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240502 08:05:45.613368 19864 tablet_service.cc:1467] Processing CreateTablet for tablet 179b4d44bff84b84b74d66c98a506f50 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.613596 19864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179b4d44bff84b84b74d66c98a506f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.613713 20373 raft_consensus.cc:348] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.613826 20373 raft_consensus.cc:374] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.613876 20373 raft_consensus.cc:729] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.613965 20372 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:45.613994 20373 consensus_queue.cc:260] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.614282 20373 ts_tablet_manager.cc:1398] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:45.614447 20373 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:45.615572 20372 tablet_bootstrap.cc:654] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.615609 20373 tablet_bootstrap.cc:654] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.615954 19863 tablet_service.cc:1467] Processing CreateTablet for tablet b44e8f73540a40d080c94498e12b1148 (DEFAULT_TABLE table=rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 08:05:45.616185 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e8f73540a40d080c94498e12b1148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:45.616204 20375 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:45.617164 20375 tablet_bootstrap.cc:654] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.617472 20372 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:45.617560 20372 ts_tablet_manager.cc:1367] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 08:05:45.618023 20372 raft_consensus.cc:348] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.618116 20372 raft_consensus.cc:374] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.618151 20372 raft_consensus.cc:729] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.618197 20373 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:45.618278 20373 ts_tablet_manager.cc:1367] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 08:05:45.618285 20372 consensus_queue.cc:260] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.618544 20372 ts_tablet_manager.cc:1398] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:45.618701 20372 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:45.618703 20373 raft_consensus.cc:348] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.618803 20373 raft_consensus.cc:374] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.618835 20373 raft_consensus.cc:729] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.619277 20375 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:45.619361 20375 ts_tablet_manager.cc:1367] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 08:05:45.619567 20372 tablet_bootstrap.cc:654] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.619740 20375 raft_consensus.cc:348] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.619844 20375 raft_consensus.cc:374] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.619881 20375 raft_consensus.cc:729] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.620005 20375 consensus_queue.cc:260] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.620180 20373 consensus_queue.cc:260] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.620585 20373 ts_tablet_manager.cc:1398] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 08:05:45.620754 20373 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:45.620774 20372 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:45.620851 20372 ts_tablet_manager.cc:1367] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240502 08:05:45.621268 20372 raft_consensus.cc:348] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.621388 20372 raft_consensus.cc:374] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.621421 20372 raft_consensus.cc:729] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.621526 20373 tablet_bootstrap.cc:654] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.621553 20372 consensus_queue.cc:260] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.621830 20372 ts_tablet_manager.cc:1398] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:45.621989 20372 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Bootstrap starting.
I20240502 08:05:45.622288 20375 ts_tablet_manager.cc:1398] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 08:05:45.622481 20375 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:45.622820 20372 tablet_bootstrap.cc:654] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.623253 20375 tablet_bootstrap.cc:654] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.624323 20373 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: No bootstrap required, opened a new log
I20240502 08:05:45.624408 20373 ts_tablet_manager.cc:1367] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:05:45.624820 20373 raft_consensus.cc:348] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.624936 20373 raft_consensus.cc:374] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.624970 20373 raft_consensus.cc:729] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.625088 20373 consensus_queue.cc:260] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.625211 20372 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: No bootstrap required, opened a new log
I20240502 08:05:45.625310 20372 ts_tablet_manager.cc:1367] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 08:05:45.625391 20373 ts_tablet_manager.cc:1398] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:45.625721 20372 raft_consensus.cc:348] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.625833 20372 raft_consensus.cc:374] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.625880 20372 raft_consensus.cc:729] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.625993 20372 consensus_queue.cc:260] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.626130 20375 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:45.626215 20375 ts_tablet_manager.cc:1367] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240502 08:05:45.626268 20372 ts_tablet_manager.cc:1398] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 08:05:45.626595 20375 raft_consensus.cc:348] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.626688 20375 raft_consensus.cc:374] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.626721 20375 raft_consensus.cc:729] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.626845 20375 consensus_queue.cc:260] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.627310 20375 ts_tablet_manager.cc:1398] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:45.627497 20375 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:45.628417 20375 tablet_bootstrap.cc:654] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Neither blocks nor log segments found. Creating new log.
I20240502 08:05:45.629642 20375 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: No bootstrap required, opened a new log
I20240502 08:05:45.629726 20375 ts_tablet_manager.cc:1367] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:45.630235 20375 raft_consensus.cc:348] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.630318 20375 raft_consensus.cc:374] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 08:05:45.630339 20375 raft_consensus.cc:729] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: FOLLOWER
I20240502 08:05:45.630453 20375 consensus_queue.cc:260] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.630720 20375 ts_tablet_manager.cc:1398] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:45.642832 20360 raft_consensus.cc:482] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:45.642958 20360 raft_consensus.cc:504] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.643391 20360 leader_election.cc:288] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:45.643491 20348 raft_consensus.cc:482] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:45.643587 20348 raft_consensus.cc:504] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.643777 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e8f73540a40d080c94498e12b1148" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:45.643769 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e8f73540a40d080c94498e12b1148" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" is_pre_election: true
I20240502 08:05:45.643877 19882 raft_consensus.cc:2454] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 0.
I20240502 08:05:45.643915 19751 raft_consensus.cc:2454] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 0.
I20240502 08:05:45.644006 20348 leader_election.cc:288] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.644215 19951 leader_election.cc:302] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, 9e4a71b756c544e4ae6c8cbe92b6eb5b; no voters:
I20240502 08:05:45.644340 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179b4d44bff84b84b74d66c98a506f50" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" is_pre_election: true
I20240502 08:05:45.644402 20360 raft_consensus.cc:2781] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:45.644493 20360 raft_consensus.cc:482] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:45.644479 19751 raft_consensus.cc:2454] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 0.
I20240502 08:05:45.644526 20360 raft_consensus.cc:3037] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.644764 19819 leader_election.cc:302] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.644793 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179b4d44bff84b84b74d66c98a506f50" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:45.644892 20016 raft_consensus.cc:2454] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 0.
I20240502 08:05:45.644953 20348 raft_consensus.cc:2781] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:45.645015 20348 raft_consensus.cc:482] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:45.645051 20348 raft_consensus.cc:3037] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.645627 20360 raft_consensus.cc:504] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.646021 20360 leader_election.cc:288] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:45.646045 20348 raft_consensus.cc:504] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.646262 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e8f73540a40d080c94498e12b1148" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:45.646323 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b44e8f73540a40d080c94498e12b1148" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:45.646364 19751 raft_consensus.cc:3037] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.646401 20348 leader_election.cc:288] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.646420 19882 raft_consensus.cc:3037] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.646795 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179b4d44bff84b84b74d66c98a506f50" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:45.646904 20016 raft_consensus.cc:3037] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.647280 19882 raft_consensus.cc:2454] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 1.
I20240502 08:05:45.647387 19751 raft_consensus.cc:2454] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 1.
I20240502 08:05:45.647596 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "179b4d44bff84b84b74d66c98a506f50" candidate_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:45.647656 19951 leader_election.cc:302] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, 9e4a71b756c544e4ae6c8cbe92b6eb5b; no voters:
I20240502 08:05:45.647687 19751 raft_consensus.cc:3037] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.647876 20016 raft_consensus.cc:2454] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 1.
I20240502 08:05:45.648059 20360 raft_consensus.cc:2781] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:45.648130 19820 leader_election.cc:302] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.648136 20360 raft_consensus.cc:686] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:45.648308 20360 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.648519 20348 raft_consensus.cc:2781] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:45.648615 20348 raft_consensus.cc:686] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Becoming Leader. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Running, Role: LEADER
I20240502 08:05:45.648772 20348 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.648586 19751 raft_consensus.cc:2454] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665e175a5ac4d2b8bffa107bfd68ec9 in term 1.
I20240502 08:05:45.650276 19603 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:45.650311 19605 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194). New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:45.656404 20218 raft_consensus.cc:482] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 08:05:45.656548 20218 raft_consensus.cc:504] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.656903 20218 leader_election.cc:288] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.657298 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" is_pre_election: true
I20240502 08:05:45.657297 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" is_pre_election: true
I20240502 08:05:45.657442 19882 raft_consensus.cc:2454] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:45.657442 20016 raft_consensus.cc:2454] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 0.
I20240502 08:05:45.657778 19685 leader_election.cc:302] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.657948 20218 raft_consensus.cc:2781] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 08:05:45.658008 20218 raft_consensus.cc:482] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 08:05:45.658056 20218 raft_consensus.cc:3037] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.658797 20218 raft_consensus.cc:504] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.659106 20218 leader_election.cc:288] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:45.659444 19882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:45.659464 20016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876" candidate_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
I20240502 08:05:45.659539 19882 raft_consensus.cc:3037] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.659571 20016 raft_consensus.cc:3037] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 0 FOLLOWER]: Advancing to term 1
I20240502 08:05:45.660261 19882 raft_consensus.cc:2454] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:45.660310 20016 raft_consensus.cc:2454] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac6fd30b17d47b99c1f0eceb5bddd6f in term 1.
I20240502 08:05:45.660563 19685 leader_election.cc:302] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:45.660785 20218 raft_consensus.cc:2781] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 08:05:45.660859 20218 raft_consensus.cc:686] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Leader. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:45.661000 20218 consensus_queue.cc:237] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:45.662575 19605 catalog_manager.cc:5484] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193). New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } }
W20240502 08:05:45.785269 20388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:45.785542 20388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:45.788641 20388 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 08:05:45.879501 20016 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.880323 19882 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.880721 20218 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.880928 20345 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.884943 20015 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.885349 20234 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.885656 19751 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.886196 20348 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.892997 19883 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.893548 20360 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 08:05:45.895432 19751 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 08:05:45.895843 20360 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.808 ms
time per row: 0.0569038 ms
I20240502 08:05:45.947631 16531 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
/tmp/dist-test-taskm5AjVx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.36.196:35635
--local_ip_for_outbound_sockets=127.16.36.196
--tserver_master_addrs=127.16.36.254:32833
--webserver_port=36455
--webserver_interface=127.16.36.196
--builtin_ntp_servers=127.16.36.212:40227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 08:05:46.057075 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 08:05:46.057322 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 08:05:46.057406 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 08:05:46.060814 20428 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 08:05:46.060899 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 08:05:46.060993 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.36.196
I20240502 08:05:46.064811 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.36.212:40227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.36.196:35635
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.16.36.196
--webserver_port=36455
--tserver_master_addrs=127.16.36.254:32833
--never_fsync=true
--heap_profile_path=/tmp/kudu.20428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.36.196
--log_dir=/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 02 May 2024 07:43:15 UTC on 1a15035a6830
build id 85
I20240502 08:05:46.065713 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 08:05:46.066813 20428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 08:05:46.075130 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:46.075703 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 08:05:46.076328 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 08:05:46.077114 20428 server_base.cc:1034] running on GCE node
I20240502 08:05:46.077476 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 08:05:46.077993 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 08:05:46.079152 20428 hybrid_clock.cc:648] HybridClock initialized: now 1714637146079136 us; error 30 us; skew 500 ppm
I20240502 08:05:46.081172 20428 webserver.cc:460] Webserver started at http://127.16.36.196:36455/ using document root <none> and password file <none>
I20240502 08:05:46.081729 20428 fs_manager.cc:359] Metadata directory not provided
I20240502 08:05:46.081809 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 08:05:46.085222 20428 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240502 08:05:46.087131 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:46.088204 20428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 08:05:46.088328 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cc250aaf098d4d10b71f6aa076a47386"
format_stamp: "Formatted at 2024-05-02 08:05:40 on dist-test-slave-9sjq"
I20240502 08:05:46.088776 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 08:05:46.101838 20428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 08:05:46.102397 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 08:05:46.102562 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 08:05:46.103068 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 08:05:46.103920 20428 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 08:05:46.103969 20428 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:46.104025 20428 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 08:05:46.104060 20428 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 08:05:46.112411 20428 rpc_server.cc:307] RPC server started. Bound to: 127.16.36.196:35635
I20240502 08:05:46.112437 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.36.196:35635 every 8 connection(s)
I20240502 08:05:46.113485 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 08:05:46.122591 20556 heartbeater.cc:347] Connected to a master server at 127.16.36.254:32833
I20240502 08:05:46.122932 20556 heartbeater.cc:464] Registering TS with master...
I20240502 08:05:46.123281 16531 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu as pid 20428
I20240502 08:05:46.123642 20556 heartbeater.cc:511] Master 127.16.36.254:32833 requested a full tablet report, sending...
I20240502 08:05:46.124897 19602 ts_manager.cc:194] Re-registered known tserver with Master: cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635)
I20240502 08:05:46.125492 19602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.36.196:50745
I20240502 08:05:46.324522 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.325037 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:46.325047 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.325533 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:46.421113 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:46.421506 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.423290 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.423954 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:46.534364 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.534854 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.535576 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:46.535672 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:46.641078 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.641171 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:46.644929 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:46.657738 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:46.695583 19882 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:46.697284 19751 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.698520 20016 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.698577 20695 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.698999 20696 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.700845 20234 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.701397 20016 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
W20240502 08:05:46.703434 19819 consensus_peers.cc:485] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 179b4d44bff84b84b74d66c98a506f50. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:46.704176 19609 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:46.705969 19751 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.714188 20016 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:46.715694 19882 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.716127 20709 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.716535 19751 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.717075 20710 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.718533 20360 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.719724 19751 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
W20240502 08:05:46.720574 19951 consensus_peers.cc:485] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9685f0c65734d7bb8bea0afe792cdd7. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:46.721194 19882 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.723395 19602 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:46.732362 19751 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:46.734355 20016 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.734918 19882 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.735397 20344 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.735612 20721 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.737885 20344 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
W20240502 08:05:46.738490 19687 consensus_peers.cc:485] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da90061bf3d9471896a2e51bb02864bb. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:46.738893 20016 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.741123 19882 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.741359 19609 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:46.751195 19751 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:46.752831 20016 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.752887 19882 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.753201 20720 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.753394 20721 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:05:46.754417 19687 consensus_peers.cc:485] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12a28c958e9d4493a112424486c56534. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:46.755336 20720 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.755926 20016 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.755884 19882 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.758611 19602 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:46.769120 19882 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:46.771131 19751 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:46.771703 20016 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 08:05:46.772008 19819 consensus_peers.cc:485] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f8d28c74e4b4be089a0d9cea5331e3c. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:46.772163 20696 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.772310 20695 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:46.774540 20695 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.775162 20016 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:46.777014 19605 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:46.787032 19752 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:47.110978 20792 ts_tablet_manager.cc:902] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:47.111728 20792 tablet_copy_client.cc:308] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:36001
I20240502 08:05:47.116000 19772 tablet_copy_service.cc:140] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Received BeginTabletCopySession request for tablet da90061bf3d9471896a2e51bb02864bb from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:33159)
I20240502 08:05:47.116163 19772 tablet_copy_service.cc:161] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Beginning new tablet copy session on tablet da90061bf3d9471896a2e51bb02864bb from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:33159: session id = cc250aaf098d4d10b71f6aa076a47386-da90061bf3d9471896a2e51bb02864bb
I20240502 08:05:47.117422 19772 tablet_copy_source_session.cc:205] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:47.120560 20792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90061bf3d9471896a2e51bb02864bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:47.125701 20792 tablet_copy_client.cc:783] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:47.125893 20792 tablet_copy_client.cc:647] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:47.127475 20556 heartbeater.cc:503] Master 127.16.36.254:32833 was elected leader, sending a full tablet report...
I20240502 08:05:47.127678 20792 tablet_copy_client.cc:523] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:47.129385 20792 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:47.134917 20792 log.cc:824] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Log is configured to *not* fsync() on all Append() calls
I20240502 08:05:47.154301 20792 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:47.154666 20792 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:47.154971 20792 ts_tablet_manager.cc:1367] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.007s
I20240502 08:05:47.157136 20792 raft_consensus.cc:348] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.157369 20792 raft_consensus.cc:729] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:47.157706 20792 consensus_queue.cc:260] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.158491 20792 ts_tablet_manager.cc:1398] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 08:05:47.159375 19772 tablet_copy_service.cc:342] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-da90061bf3d9471896a2e51bb02864bb received from {username='slave'} at 127.16.36.196:33159
I20240502 08:05:47.159475 19772 tablet_copy_service.cc:433] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-da90061bf3d9471896a2e51bb02864bb on tablet da90061bf3d9471896a2e51bb02864bb with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:47.199527 20792 ts_tablet_manager.cc:902] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:47.200017 20792 tablet_copy_client.cc:308] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:45299
I20240502 08:05:47.203874 19904 tablet_copy_service.cc:140] P a665e175a5ac4d2b8bffa107bfd68ec9: Received BeginTabletCopySession request for tablet 179b4d44bff84b84b74d66c98a506f50 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:51827)
I20240502 08:05:47.203975 20804 ts_tablet_manager.cc:902] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:47.204030 19904 tablet_copy_service.cc:161] P a665e175a5ac4d2b8bffa107bfd68ec9: Beginning new tablet copy session on tablet 179b4d44bff84b84b74d66c98a506f50 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:51827: session id = cc250aaf098d4d10b71f6aa076a47386-179b4d44bff84b84b74d66c98a506f50
I20240502 08:05:47.204511 20804 tablet_copy_client.cc:308] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:36001
I20240502 08:05:47.204948 19772 tablet_copy_service.cc:140] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Received BeginTabletCopySession request for tablet 12a28c958e9d4493a112424486c56534 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:33159)
I20240502 08:05:47.205039 19772 tablet_copy_service.cc:161] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Beginning new tablet copy session on tablet 12a28c958e9d4493a112424486c56534 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:33159: session id = cc250aaf098d4d10b71f6aa076a47386-12a28c958e9d4493a112424486c56534
I20240502 08:05:47.205449 19904 tablet_copy_source_session.cc:205] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:47.205734 19772 tablet_copy_source_session.cc:205] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:47.206598 20792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179b4d44bff84b84b74d66c98a506f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:47.206813 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a28c958e9d4493a112424486c56534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:47.208503 20804 tablet_copy_client.cc:783] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:47.208657 20804 tablet_copy_client.cc:647] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:47.208801 20792 tablet_copy_client.cc:783] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:47.208940 20792 tablet_copy_client.cc:647] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:47.209939 20804 tablet_copy_client.cc:523] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:47.210254 20792 tablet_copy_client.cc:523] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:47.210968 20804 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:47.211221 20792 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:47.229140 20792 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:47.229219 20804 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:47.229427 20792 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:47.229488 20804 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:47.229609 20792 ts_tablet_manager.cc:1367] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.018s user 0.017s sys 0.000s
I20240502 08:05:47.229657 20804 ts_tablet_manager.cc:1367] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.006s
I20240502 08:05:47.230175 20804 raft_consensus.cc:348] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.230191 20792 raft_consensus.cc:348] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.230312 20804 raft_consensus.cc:729] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:47.230324 20792 raft_consensus.cc:729] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:47.230453 20804 consensus_queue.cc:260] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.230456 20792 consensus_queue.cc:260] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.230820 20792 ts_tablet_manager.cc:1398] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 08:05:47.231218 20804 ts_tablet_manager.cc:1398] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.002s sys 0.001s
I20240502 08:05:47.231578 19904 tablet_copy_service.cc:342] P a665e175a5ac4d2b8bffa107bfd68ec9: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-179b4d44bff84b84b74d66c98a506f50 received from {username='slave'} at 127.16.36.196:51827
I20240502 08:05:47.231678 19904 tablet_copy_service.cc:433] P a665e175a5ac4d2b8bffa107bfd68ec9: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-179b4d44bff84b84b74d66c98a506f50 on tablet 179b4d44bff84b84b74d66c98a506f50 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:47.231732 19772 tablet_copy_service.cc:342] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-12a28c958e9d4493a112424486c56534 received from {username='slave'} at 127.16.36.196:33159
I20240502 08:05:47.231806 19772 tablet_copy_service.cc:433] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-12a28c958e9d4493a112424486c56534 on tablet 12a28c958e9d4493a112424486c56534 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:47.295254 20792 ts_tablet_manager.cc:902] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:47.295783 20792 tablet_copy_client.cc:308] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:45299
I20240502 08:05:47.296252 19904 tablet_copy_service.cc:140] P a665e175a5ac4d2b8bffa107bfd68ec9: Received BeginTabletCopySession request for tablet 3f8d28c74e4b4be089a0d9cea5331e3c from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:51827)
I20240502 08:05:47.296371 19904 tablet_copy_service.cc:161] P a665e175a5ac4d2b8bffa107bfd68ec9: Beginning new tablet copy session on tablet 3f8d28c74e4b4be089a0d9cea5331e3c from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:51827: session id = cc250aaf098d4d10b71f6aa076a47386-3f8d28c74e4b4be089a0d9cea5331e3c
I20240502 08:05:47.297333 19904 tablet_copy_source_session.cc:205] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:47.298177 20792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d28c74e4b4be089a0d9cea5331e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:47.299923 20792 tablet_copy_client.cc:783] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:47.300069 20792 tablet_copy_client.cc:647] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:47.301157 20792 tablet_copy_client.cc:523] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:47.302119 20792 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:47.303443 20804 ts_tablet_manager.cc:902] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:47.304008 20804 tablet_copy_client.cc:308] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:44147
I20240502 08:05:47.307838 20036 tablet_copy_service.cc:140] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Received BeginTabletCopySession request for tablet f9685f0c65734d7bb8bea0afe792cdd7 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:44923)
I20240502 08:05:47.307969 20036 tablet_copy_service.cc:161] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Beginning new tablet copy session on tablet f9685f0c65734d7bb8bea0afe792cdd7 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:44923: session id = cc250aaf098d4d10b71f6aa076a47386-f9685f0c65734d7bb8bea0afe792cdd7
I20240502 08:05:47.309144 20036 tablet_copy_source_session.cc:205] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:47.310196 20804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9685f0c65734d7bb8bea0afe792cdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:47.312089 20804 tablet_copy_client.cc:783] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:47.312240 20804 tablet_copy_client.cc:647] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:47.313545 20804 tablet_copy_client.cc:523] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:47.314500 20804 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:47.321828 20792 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:47.322086 20792 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:47.322247 20792 ts_tablet_manager.cc:1367] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240502 08:05:47.322667 20792 raft_consensus.cc:348] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.322788 20792 raft_consensus.cc:729] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:47.322887 20792 consensus_queue.cc:260] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.323110 20792 ts_tablet_manager.cc:1398] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:47.323961 19904 tablet_copy_service.cc:342] P a665e175a5ac4d2b8bffa107bfd68ec9: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-3f8d28c74e4b4be089a0d9cea5331e3c received from {username='slave'} at 127.16.36.196:51827
I20240502 08:05:47.324038 19904 tablet_copy_service.cc:433] P a665e175a5ac4d2b8bffa107bfd68ec9: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-3f8d28c74e4b4be089a0d9cea5331e3c on tablet 3f8d28c74e4b4be089a0d9cea5331e3c with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:47.332815 20804 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:47.333096 20804 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:47.333264 20804 ts_tablet_manager.cc:1367] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.019s user 0.018s sys 0.000s
I20240502 08:05:47.333716 20804 raft_consensus.cc:348] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.333832 20804 raft_consensus.cc:729] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:47.333935 20804 consensus_queue.cc:260] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:47.334187 20804 ts_tablet_manager.cc:1398] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:47.334935 20036 tablet_copy_service.cc:342] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-f9685f0c65734d7bb8bea0afe792cdd7 received from {username='slave'} at 127.16.36.196:44923
I20240502 08:05:47.335060 20036 tablet_copy_service.cc:433] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-f9685f0c65734d7bb8bea0afe792cdd7 on tablet f9685f0c65734d7bb8bea0afe792cdd7 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:47.601629 20510 raft_consensus.cc:1203] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:47.658589 20510 raft_consensus.cc:1203] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:47.695890 20510 raft_consensus.cc:1203] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:47.736809 20510 raft_consensus.cc:1203] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:47.818574 20510 raft_consensus.cc:1203] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:48.138299 20725 raft_consensus.cc:1050] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:48.138793 20725 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.140288 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.140287 20016 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.140308 20510 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.140753 20725 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.140966 20872 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.141242 20929 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.143838 20725 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.144348 20016 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.144352 20510 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.146628 19883 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.147321 19609 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.151768 19752 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.153050 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.153064 20510 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.153443 20929 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.153642 20873 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.155216 20872 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.155627 20510 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.155836 19883 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.157277 19596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da90061bf3d9471896a2e51bb02864bb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:48.158197 19605 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.160951 19995 tablet_service.cc:1514] Processing DeleteTablet for tablet da90061bf3d9471896a2e51bb02864bb with delete_type TABLET_DATA_TOMBSTONED (TS 9e4a71b756c544e4ae6c8cbe92b6eb5b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38732
I20240502 08:05:48.161562 20937 tablet_replica.cc:330] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: stopping tablet replica
I20240502 08:05:48.161852 20937 raft_consensus.cc:2229] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:48.162081 20937 raft_consensus.cc:2258] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:48.163283 20937 ts_tablet_manager.cc:1875] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:48.166733 20937 ts_tablet_manager.cc:1888] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:48.166855 20937 log.cc:1196] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/da90061bf3d9471896a2e51bb02864bb
I20240502 08:05:48.167723 19597 catalog_manager.cc:4830] TS 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147): tablet da90061bf3d9471896a2e51bb02864bb (table rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]) successfully deleted
I20240502 08:05:48.189730 20874 raft_consensus.cc:1050] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:48.190341 20874 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.192168 20510 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.192231 19752 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.192675 20954 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.192721 20016 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.192878 20234 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.193704 20954 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.195969 20234 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.196318 20016 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.196985 20510 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.199357 19602 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.201036 19752 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.205082 19883 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.206267 19752 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.206498 20872 raft_consensus.cc:1050] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:48.206503 20510 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.206789 20963 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.206934 20872 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.207119 20963 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.208631 19883 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.208789 20509 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.208819 20016 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.208868 20954 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.209003 20725 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.209268 20929 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.210978 19594 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f8d28c74e4b4be089a0d9cea5331e3c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:48.211230 20725 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.211457 19752 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.211742 19609 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:48.214505 20510 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.215909 20929 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.216522 20510 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.218003 19995 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8d28c74e4b4be089a0d9cea5331e3c with delete_type TABLET_DATA_TOMBSTONED (TS 9e4a71b756c544e4ae6c8cbe92b6eb5b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38732
I20240502 08:05:48.217949 20016 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.218199 20937 tablet_replica.cc:330] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: stopping tablet replica
I20240502 08:05:48.218377 20937 raft_consensus.cc:2229] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:48.218528 20937 raft_consensus.cc:2258] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:48.220259 19606 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:48.220444 20937 ts_tablet_manager.cc:1875] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:48.222705 20937 ts_tablet_manager.cc:1888] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:48.222791 20937 log.cc:1196] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/3f8d28c74e4b4be089a0d9cea5331e3c
I20240502 08:05:48.223277 19597 catalog_manager.cc:4830] TS 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147): tablet 3f8d28c74e4b4be089a0d9cea5331e3c (table rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]) successfully deleted
I20240502 08:05:48.223574 19752 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.224720 20510 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.224756 20016 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.225054 20873 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.225539 20725 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.226701 20902 raft_consensus.cc:1050] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:48.226972 20875 raft_consensus.cc:1050] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:48.227082 20873 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.227169 20902 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.227403 20875 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.227473 20510 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.228915 20509 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.229261 20901 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.229522 19883 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.229784 19605 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:48.230103 19596 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12a28c958e9d4493a112424486c56534 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:48.230324 20510 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.230372 19752 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.230425 19751 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.230684 20234 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.230916 20901 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.230911 20016 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:48.231557 20901 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.231940 20234 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.232281 20874 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:48.232300 20016 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.233852 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a28c958e9d4493a112424486c56534 with delete_type TABLET_DATA_TOMBSTONED (TS a665e175a5ac4d2b8bffa107bfd68ec9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50238
I20240502 08:05:48.235249 20986 tablet_replica.cc:330] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: stopping tablet replica
I20240502 08:05:48.235559 20901 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.236868 19752 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.237453 19751 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.236347 20234 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.237569 20509 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.237157 19883 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.240361 20015 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.240411 19609 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.235632 20986 raft_consensus.cc:2229] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:48.243590 20015 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.243891 19605 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.244956 20509 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.245092 20986 pending_rounds.cc:62] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Trying to abort 1 pending ops.
I20240502 08:05:48.245312 20015 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:48.245365 20979 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.245335 20986 pending_rounds.cc:68] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023153759054389248 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "12a28c958e9d4493a112424486c56534" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } } }
I20240502 08:05:48.245748 20986 raft_consensus.cc:2894] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 08:05:48.246054 19751 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.246227 20986 raft_consensus.cc:2866] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 08:05:48.246347 20986 raft_consensus.cc:2258] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:48.247670 19883 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:48.247887 20508 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.248059 20979 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 08:05:48.248713 19951 consensus_peers.cc:485] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b44e8f73540a40d080c94498e12b1148. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:48.248999 20979 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.249204 19751 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:48.249553 20979 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:48.250995 20709 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:48.251466 19883 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:48.251807 19882 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:48.251786 20902 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.253649 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9685f0c65734d7bb8bea0afe792cdd7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:48.253641 19751 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.254096 20508 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.254410 19750 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:48.254580 20963 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.254650 19602 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:48.254828 20874 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:48.257128 19606 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.257524 20986 ts_tablet_manager.cc:1875] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:48.257485 20509 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.259069 20874 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.259943 20986 ts_tablet_manager.cc:1888] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:48.260038 20986 log.cc:1196] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/12a28c958e9d4493a112424486c56534
I20240502 08:05:48.261049 19594 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 179b4d44bff84b84b74d66c98a506f50 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:48.261210 19594 catalog_manager.cc:4830] TS a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299): tablet 12a28c958e9d4493a112424486c56534 (table rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]) successfully deleted
I20240502 08:05:48.261289 19752 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.261518 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet f9685f0c65734d7bb8bea0afe792cdd7 with delete_type TABLET_DATA_TOMBSTONED (TS a665e175a5ac4d2b8bffa107bfd68ec9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50238
I20240502 08:05:48.262220 20509 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:48.263068 20986 tablet_replica.cc:330] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: stopping tablet replica
I20240502 08:05:48.263221 20986 raft_consensus.cc:2229] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:48.263381 20986 raft_consensus.cc:2258] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:48.263638 19602 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 5 to 6, VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) evicted. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:48.264122 20986 ts_tablet_manager.cc:1875] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:48.264837 19751 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:48.266162 19995 tablet_service.cc:1514] Processing DeleteTablet for tablet 179b4d44bff84b84b74d66c98a506f50 with delete_type TABLET_DATA_TOMBSTONED (TS 9e4a71b756c544e4ae6c8cbe92b6eb5b not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:38732
I20240502 08:05:48.266399 20937 tablet_replica.cc:330] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: stopping tablet replica
I20240502 08:05:48.266552 20937 raft_consensus.cc:2229] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:48.266739 20937 raft_consensus.cc:2258] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:48.267309 20986 ts_tablet_manager.cc:1888] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:48.267395 20986 log.cc:1196] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9685f0c65734d7bb8bea0afe792cdd7
I20240502 08:05:48.267743 20937 ts_tablet_manager.cc:1875] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:48.267864 19594 catalog_manager.cc:4830] TS a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299): tablet f9685f0c65734d7bb8bea0afe792cdd7 (table rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]) successfully deleted
I20240502 08:05:48.270025 20937 ts_tablet_manager.cc:1888] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:48.270098 20937 log.cc:1196] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-2/wal/wals/179b4d44bff84b84b74d66c98a506f50
I20240502 08:05:48.270629 19597 catalog_manager.cc:4830] TS 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147): tablet 179b4d44bff84b84b74d66c98a506f50 (table rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]) successfully deleted
I20240502 08:05:48.707281 21026 ts_tablet_manager.cc:902] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:48.707801 21026 tablet_copy_client.cc:308] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:44147
I20240502 08:05:48.708317 20036 tablet_copy_service.cc:140] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Received BeginTabletCopySession request for tablet b44e8f73540a40d080c94498e12b1148 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:44923)
I20240502 08:05:48.708425 20036 tablet_copy_service.cc:161] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Beginning new tablet copy session on tablet b44e8f73540a40d080c94498e12b1148 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:44923: session id = cc250aaf098d4d10b71f6aa076a47386-b44e8f73540a40d080c94498e12b1148
I20240502 08:05:48.709338 20036 tablet_copy_source_session.cc:205] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:48.710250 21026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e8f73540a40d080c94498e12b1148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:48.712273 21026 tablet_copy_client.cc:783] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:48.712482 21026 tablet_copy_client.cc:647] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:48.713734 21026 tablet_copy_client.cc:523] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:48.714808 21026 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:48.731499 21026 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:48.731777 21026 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:48.731922 21026 ts_tablet_manager.cc:1367] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20240502 08:05:48.732302 21026 raft_consensus.cc:348] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:48.732409 21026 raft_consensus.cc:729] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:48.732527 21026 consensus_queue.cc:260] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:48.732748 21026 ts_tablet_manager.cc:1398] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 08:05:48.733307 20036 tablet_copy_service.cc:342] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-b44e8f73540a40d080c94498e12b1148 received from {username='slave'} at 127.16.36.196:44923
I20240502 08:05:48.733390 20036 tablet_copy_service.cc:433] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-b44e8f73540a40d080c94498e12b1148 on tablet b44e8f73540a40d080c94498e12b1148 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:49.124110 20509 raft_consensus.cc:1203] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 08:05:49.626139 20902 raft_consensus.cc:1050] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:49.626514 20902 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:49.627904 19883 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:49.627954 20509 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:49.627954 19751 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:49.628371 21060 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:49.628636 21061 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:49.628839 21058 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:49.630697 21061 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.631184 20509 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.631194 19883 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.632500 19751 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.633828 19609 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 4 to 5, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:49.637271 20015 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:49.638478 20509 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:49.638492 19751 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:49.638883 21058 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:49.639200 21060 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:49.640875 20902 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.642128 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b44e8f73540a40d080c94498e12b1148 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:49.642520 20509 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.643070 19606 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:49.644093 19751 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:49.646497 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet b44e8f73540a40d080c94498e12b1148 with delete_type TABLET_DATA_TOMBSTONED (TS a665e175a5ac4d2b8bffa107bfd68ec9 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:50238
I20240502 08:05:49.646888 21070 tablet_replica.cc:330] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: stopping tablet replica
I20240502 08:05:49.647166 21070 raft_consensus.cc:2229] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:49.647387 21070 raft_consensus.cc:2258] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:49.648620 21070 ts_tablet_manager.cc:1875] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:49.650377 21070 ts_tablet_manager.cc:1888] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 08:05:49.650444 21070 log.cc:1196] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-1/wal/wals/b44e8f73540a40d080c94498e12b1148
I20240502 08:05:49.650964 19594 catalog_manager.cc:4830] TS a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299): tablet b44e8f73540a40d080c94498e12b1148 (table rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]) successfully deleted
I20240502 08:05:49.800966 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:49.801074 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:49.801841 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:49.802552 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:49.893242 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:49.894033 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:49.895233 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:49.904636 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:49.979674 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:49.980084 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:49.980716 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:49.981588 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.161192 19605 ts_manager.cc:292] Set tserver state for 8ac6fd30b17d47b99c1f0eceb5bddd6f to MAINTENANCE_MODE
I20240502 08:05:50.352510 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:50.353256 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.353919 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.355036 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:50.444919 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:50.445256 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.445520 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.446030 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:50.552716 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:50.553647 19995 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.554406 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:50.555058 19732 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 08:05:50.589612 19751 consensus_queue.cc:237] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:50.590780 20015 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:50.590833 19883 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:50.591241 21289 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:50.591436 21172 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:50.593403 21292 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:50.593676 19883 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:50.593688 20015 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:50.596901 19602 catalog_manager.cc:5484] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:50.601276 19751 consensus_queue.cc:237] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:50.602458 20015 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:05:50.602658 19687 consensus_peers.cc:485] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ee8e3ede5eb4640bbb530277175dd52. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:50.602752 19883 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:50.602845 21172 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:50.603176 21172 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:50.604882 21289 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:50.605140 20015 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:50.606482 19596 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0ee8e3ede5eb4640bbb530277175dd52 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:50.606519 19883 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:50.607277 19606 catalog_manager.cc:5484] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:50.610307 20015 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:50.611421 19751 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:50.611429 20509 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:50.611793 21299 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:50.611969 20902 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:50.613816 21302 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.614184 19751 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.614195 20509 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.616392 19602 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.619899 20015 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:50.621362 20509 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:50.621364 19751 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:50.621724 21299 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:05:50.622027 19949 consensus_peers.cc:485] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b -> Peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299): Couldn't send request to peer a665e175a5ac4d2b8bffa107bfd68ec9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:50.622155 21299 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:50.623674 21302 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:50.624578 19752 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:50.625098 19597 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f9685f0c65734d7bb8bea0afe792cdd7 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:50.625532 19752 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:50.625613 20509 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:50.625810 20508 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:50.626034 19606 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:50.626593 21172 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:50.627388 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:50.627743 21289 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:50.629274 21289 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.630753 19883 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.631105 20509 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:50.631507 19606 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:50.634943 19752 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:50.636261 20509 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:50.636391 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:05:50.636606 19688 consensus_peers.cc:485] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147): Couldn't send request to peer 9e4a71b756c544e4ae6c8cbe92b6eb5b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:50.636751 21172 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:50.636942 21289 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:50.638828 19883 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:50.638540 21172 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:50.640209 19596 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet da90061bf3d9471896a2e51bb02864bb with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:50.641057 20509 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:50.641407 19602 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:51.030947 21342 ts_tablet_manager.cc:902] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Initiating tablet copy from peer 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:51.031810 21342 tablet_copy_client.cc:308] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:36001
I20240502 08:05:51.032363 19772 tablet_copy_service.cc:140] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Received BeginTabletCopySession request for tablet da90061bf3d9471896a2e51bb02864bb from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b ({username='slave'} at 127.16.36.195:38069)
I20240502 08:05:51.032508 19772 tablet_copy_service.cc:161] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Beginning new tablet copy session on tablet da90061bf3d9471896a2e51bb02864bb from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b at {username='slave'} at 127.16.36.195:38069: session id = 9e4a71b756c544e4ae6c8cbe92b6eb5b-da90061bf3d9471896a2e51bb02864bb
I20240502 08:05:51.033246 19772 tablet_copy_source_session.cc:205] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:51.034142 21342 ts_tablet_manager.cc:1875] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:51.036106 21342 ts_tablet_manager.cc:1888] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:51.036262 21342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90061bf3d9471896a2e51bb02864bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.037308 21342 tablet_copy_client.cc:783] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:51.037484 21342 tablet_copy_client.cc:647] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:51.038865 21342 tablet_copy_client.cc:523] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:51.040146 21342 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:51.059782 21342 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:51.060204 21342 tablet_bootstrap.cc:492] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap complete.
I20240502 08:05:51.060425 21342 ts_tablet_manager.cc:1367] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.003s
I20240502 08:05:51.060881 21342 raft_consensus.cc:348] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:51.060993 21342 raft_consensus.cc:729] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: LEARNER
I20240502 08:05:51.061081 21342 consensus_queue.cc:260] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:51.061363 21342 ts_tablet_manager.cc:1398] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:51.062034 19772 tablet_copy_service.cc:342] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Request end of tablet copy session 9e4a71b756c544e4ae6c8cbe92b6eb5b-da90061bf3d9471896a2e51bb02864bb received from {username='slave'} at 127.16.36.195:38069
I20240502 08:05:51.062115 19772 tablet_copy_service.cc:433] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: ending tablet copy session 9e4a71b756c544e4ae6c8cbe92b6eb5b-da90061bf3d9471896a2e51bb02864bb on tablet da90061bf3d9471896a2e51bb02864bb with peer 9e4a71b756c544e4ae6c8cbe92b6eb5b
I20240502 08:05:51.131083 21363 ts_tablet_manager.cc:902] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Initiating tablet copy from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:51.131796 21363 tablet_copy_client.cc:308] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:44147
I20240502 08:05:51.132403 20036 tablet_copy_service.cc:140] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Received BeginTabletCopySession request for tablet f9685f0c65734d7bb8bea0afe792cdd7 from peer a665e175a5ac4d2b8bffa107bfd68ec9 ({username='slave'} at 127.16.36.194:49229)
I20240502 08:05:51.132537 20036 tablet_copy_service.cc:161] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Beginning new tablet copy session on tablet f9685f0c65734d7bb8bea0afe792cdd7 from peer a665e175a5ac4d2b8bffa107bfd68ec9 at {username='slave'} at 127.16.36.194:49229: session id = a665e175a5ac4d2b8bffa107bfd68ec9-f9685f0c65734d7bb8bea0afe792cdd7
I20240502 08:05:51.133240 20036 tablet_copy_source_session.cc:205] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:51.134152 21363 ts_tablet_manager.cc:1875] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:51.135934 21363 ts_tablet_manager.cc:1888] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:51.136111 21363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9685f0c65734d7bb8bea0afe792cdd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.137130 21363 tablet_copy_client.cc:783] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:51.137281 21363 tablet_copy_client.cc:647] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:51.138595 21363 tablet_copy_client.cc:523] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:51.139819 21363 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:51.149555 21369 ts_tablet_manager.cc:902] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:51.150094 21369 tablet_copy_client.cc:308] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:36001
I20240502 08:05:51.150610 19772 tablet_copy_service.cc:140] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Received BeginTabletCopySession request for tablet 0ee8e3ede5eb4640bbb530277175dd52 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:33159)
I20240502 08:05:51.150738 19772 tablet_copy_service.cc:161] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Beginning new tablet copy session on tablet 0ee8e3ede5eb4640bbb530277175dd52 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:33159: session id = cc250aaf098d4d10b71f6aa076a47386-0ee8e3ede5eb4640bbb530277175dd52
I20240502 08:05:51.151666 19772 tablet_copy_source_session.cc:205] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:51.152571 21369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee8e3ede5eb4640bbb530277175dd52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:51.154461 21369 tablet_copy_client.cc:783] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:51.154608 21369 tablet_copy_client.cc:647] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:51.155690 21369 tablet_copy_client.cc:523] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:51.156972 21369 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:51.162585 21363 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:51.162917 21363 tablet_bootstrap.cc:492] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap complete.
I20240502 08:05:51.163094 21363 ts_tablet_manager.cc:1367] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.023s user 0.022s sys 0.000s
I20240502 08:05:51.163631 21363 raft_consensus.cc:348] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:51.163767 21363 raft_consensus.cc:729] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: LEARNER
I20240502 08:05:51.163902 21363 consensus_queue.cc:260] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:51.164283 21363 ts_tablet_manager.cc:1398] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 08:05:51.167209 20036 tablet_copy_service.cc:342] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Request end of tablet copy session a665e175a5ac4d2b8bffa107bfd68ec9-f9685f0c65734d7bb8bea0afe792cdd7 received from {username='slave'} at 127.16.36.194:49229
I20240502 08:05:51.167296 20036 tablet_copy_service.cc:433] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: ending tablet copy session a665e175a5ac4d2b8bffa107bfd68ec9-f9685f0c65734d7bb8bea0afe792cdd7 on tablet f9685f0c65734d7bb8bea0afe792cdd7 with peer a665e175a5ac4d2b8bffa107bfd68ec9
I20240502 08:05:51.180366 21369 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:51.180930 21369 tablet_bootstrap.cc:492] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:51.181097 21369 ts_tablet_manager.cc:1367] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.007s
I20240502 08:05:51.181504 21369 raft_consensus.cc:348] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:51.181622 21369 raft_consensus.cc:729] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:51.181730 21369 consensus_queue.cc:260] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:51.182070 21369 ts_tablet_manager.cc:1398] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 08:05:51.182744 19772 tablet_copy_service.cc:342] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-0ee8e3ede5eb4640bbb530277175dd52 received from {username='slave'} at 127.16.36.196:33159
I20240502 08:05:51.182843 19772 tablet_copy_service.cc:433] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-0ee8e3ede5eb4640bbb530277175dd52 on tablet 0ee8e3ede5eb4640bbb530277175dd52 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:51.519991 20015 raft_consensus.cc:1203] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:05:51.638768 20509 raft_consensus.cc:1203] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:05:51.654960 19883 raft_consensus.cc:1203] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:05:52.081878 21289 raft_consensus.cc:1050] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: attempting to promote NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b to VOTER
I20240502 08:05:52.082361 21289 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:52.083752 20509 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:52.083752 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:52.084048 20015 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:52.084169 21441 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:52.084408 21402 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:52.084779 21401 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:52.086674 21402 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:52.087131 19883 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:52.087255 20509 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:52.089061 20015 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:52.089252 19605 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:52.173960 21441 raft_consensus.cc:1050] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:52.174386 21441 consensus_queue.cc:237] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:52.176071 19883 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:52.176071 20015 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:52.176208 20509 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:52.176560 21401 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:52.176740 21289 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:52.176952 21402 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:52.179277 21401 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:52.179749 20509 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:52.181504 20015 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:52.181874 19883 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:52.182627 19602 catalog_manager.cc:5484] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:52.185546 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40314
I20240502 08:05:52.185740 19752 raft_consensus.cc:595] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:52.197577 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40324
I20240502 08:05:52.197741 19752 raft_consensus.cc:595] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:52.234814 21478 raft_consensus.cc:1050] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: attempting to promote NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 to VOTER
I20240502 08:05:52.235230 21478 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:52.236778 20509 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:52.236768 19883 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:52.236788 19752 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:52.237213 21480 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:52.237435 21478 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:52.237613 21409 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:52.239631 21409 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.240229 20509 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.240288 19752 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.241691 19883 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.242741 19606 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.245893 20015 consensus_queue.cc:237] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:52.247030 19883 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:52.247082 20509 raft_consensus.cc:1261] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:52.247520 21480 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:52.247752 21481 consensus_queue.cc:1035] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:52.249492 21481 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.249940 19883 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.250073 20509 raft_consensus.cc:2932] T f9685f0c65734d7bb8bea0afe792cdd7 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:52.251775 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9685f0c65734d7bb8bea0afe792cdd7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:52.252127 19602 catalog_manager.cc:5484] T f9685f0c65734d7bb8bea0afe792cdd7 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:52.254930 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet f9685f0c65734d7bb8bea0afe792cdd7 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:52.255550 21485 tablet_replica.cc:330] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:52.255890 21485 raft_consensus.cc:2229] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:52.256145 21485 raft_consensus.cc:2258] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:52.257261 21485 ts_tablet_manager.cc:1875] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:52.259264 21485 ts_tablet_manager.cc:1888] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:05:52.259353 21485 log.cc:1196] T f9685f0c65734d7bb8bea0afe792cdd7 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9685f0c65734d7bb8bea0afe792cdd7
I20240502 08:05:52.259967 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet f9685f0c65734d7bb8bea0afe792cdd7 (table rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]) successfully deleted
I20240502 08:05:52.416735 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40344
I20240502 08:05:52.416878 19752 raft_consensus.cc:595] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:52.427791 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40352
I20240502 08:05:52.427932 19752 raft_consensus.cc:595] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:52.529103 21402 raft_consensus.cc:979] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: : Instructing follower 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:52.529222 21402 raft_consensus.cc:1067] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Signalling peer 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:52.529847 20015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb"
dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
from {username='slave'} at 127.16.36.193:57099
I20240502 08:05:52.529996 20015 raft_consensus.cc:482] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:52.530057 20015 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.530900 20015 raft_consensus.cc:504] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:52.531349 20015 leader_election.cc:288] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635)
I20240502 08:05:52.531667 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:52.531808 19752 raft_consensus.cc:3032] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:52.531795 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:52.531859 19752 raft_consensus.cc:729] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:52.531920 19883 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.531894 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90061bf3d9471896a2e51bb02864bb" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "cc250aaf098d4d10b71f6aa076a47386"
I20240502 08:05:52.532025 19752 consensus_queue.cc:260] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:52.532068 20509 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.532250 19752 raft_consensus.cc:3037] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.532893 19883 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:52.533169 19752 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:52.533172 20509 raft_consensus.cc:2454] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:52.533488 19951 leader_election.cc:302] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9, cc250aaf098d4d10b71f6aa076a47386; no voters:
I20240502 08:05:52.533687 21480 raft_consensus.cc:2781] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:52.533782 21480 raft_consensus.cc:686] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:52.533953 21480 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:52.535679 19605 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 1 to 2, leader changed from 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:52.629194 21402 raft_consensus.cc:979] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: : Instructing follower 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:52.629302 21402 raft_consensus.cc:1067] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Signalling peer 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:52.629832 20015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52"
dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
from {username='slave'} at 127.16.36.193:57099
I20240502 08:05:52.629985 20015 raft_consensus.cc:482] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:52.630067 20015 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.630982 20015 raft_consensus.cc:504] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:52.631438 20015 leader_election.cc:288] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Requested vote from peers a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635)
I20240502 08:05:52.631855 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:52.631877 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "cc250aaf098d4d10b71f6aa076a47386"
I20240502 08:05:52.631916 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee8e3ede5eb4640bbb530277175dd52" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:52.632000 19883 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.632022 20509 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.632027 19752 raft_consensus.cc:3032] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:52.632076 19752 raft_consensus.cc:729] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:52.632244 19752 consensus_queue.cc:260] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:52.632488 19752 raft_consensus.cc:3037] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:52.633080 20509 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:52.633093 19883 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:52.633481 19951 leader_election.cc:302] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9, cc250aaf098d4d10b71f6aa076a47386; no voters:
I20240502 08:05:52.633514 19752 raft_consensus.cc:2454] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:52.633714 21480 raft_consensus.cc:2781] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:52.633857 21480 raft_consensus.cc:686] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:52.634012 21480 consensus_queue.cc:237] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:52.635879 19609 catalog_manager.cc:5484] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 1 to 2, leader changed from 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:52.641732 19752 consensus_queue.cc:237] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:52.643082 19883 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:52.643095 20015 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:52.643527 21289 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:52.643755 21402 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:52.645769 21289 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:52.646461 19883 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:52.647761 20015 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:52.649443 19609 catalog_manager.cc:5484] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } }
I20240502 08:05:52.652776 19752 consensus_queue.cc:237] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:52.654204 19883 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 08:05:52.654757 19687 consensus_peers.cc:485] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bf541bb2ff945a386808e1b025cfec0. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:52.654877 20015 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:52.654843 21401 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:52.655400 21401 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:52.656772 21402 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:52.657462 20015 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:52.658497 19596 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5bf541bb2ff945a386808e1b025cfec0 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:52.659108 19602 catalog_manager.cc:5484] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:52.659065 19883 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:52.997110 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:52.997606 21480 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:53.054550 20509 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:53.055018 21480 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.084514 21550 ts_tablet_manager.cc:902] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:53.085072 21550 tablet_copy_client.cc:308] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:36001
I20240502 08:05:53.085628 19772 tablet_copy_service.cc:140] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Received BeginTabletCopySession request for tablet 5bf541bb2ff945a386808e1b025cfec0 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:33159)
I20240502 08:05:53.085747 19772 tablet_copy_service.cc:161] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Beginning new tablet copy session on tablet 5bf541bb2ff945a386808e1b025cfec0 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:33159: session id = cc250aaf098d4d10b71f6aa076a47386-5bf541bb2ff945a386808e1b025cfec0
I20240502 08:05:53.086531 19772 tablet_copy_source_session.cc:205] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:53.087399 21550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf541bb2ff945a386808e1b025cfec0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:53.089248 21550 tablet_copy_client.cc:783] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:53.089391 21550 tablet_copy_client.cc:647] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:53.090550 21550 tablet_copy_client.cc:523] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:53.091666 21550 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:53.094686 20509 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:53.095115 21480 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:53.098860 19752 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:53.099272 21480 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:53.102602 20015 consensus_queue.cc:237] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:53.103901 20509 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:05:53.103909 19883 raft_consensus.cc:1261] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:05:53.104357 21558 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:53.104550 21481 consensus_queue.cc:1035] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:53.106245 21481 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:53.106977 20509 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:53.107816 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da90061bf3d9471896a2e51bb02864bb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:53.109136 19883 raft_consensus.cc:2932] T da90061bf3d9471896a2e51bb02864bb P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:53.109316 19602 catalog_manager.cc:5484] T da90061bf3d9471896a2e51bb02864bb P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:53.111850 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet da90061bf3d9471896a2e51bb02864bb with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:53.121093 21568 tablet_replica.cc:330] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:53.121281 21568 raft_consensus.cc:2229] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:53.121441 21568 raft_consensus.cc:2258] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:53.121693 21550 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:53.122020 21550 tablet_bootstrap.cc:492] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:53.122205 21550 ts_tablet_manager.cc:1367] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.003s
I20240502 08:05:53.122627 21550 raft_consensus.cc:348] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:53.122764 21550 raft_consensus.cc:729] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:53.122854 21550 consensus_queue.cc:260] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:53.123099 21550 ts_tablet_manager.cc:1398] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:53.123844 19772 tablet_copy_service.cc:342] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-5bf541bb2ff945a386808e1b025cfec0 received from {username='slave'} at 127.16.36.196:33159
I20240502 08:05:53.123925 19772 tablet_copy_service.cc:433] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-5bf541bb2ff945a386808e1b025cfec0 on tablet 5bf541bb2ff945a386808e1b025cfec0 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:53.124085 21568 ts_tablet_manager.cc:1875] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:53.126838 21568 ts_tablet_manager.cc:1888] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:05:53.126926 21568 log.cc:1196] T da90061bf3d9471896a2e51bb02864bb P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/da90061bf3d9471896a2e51bb02864bb
I20240502 08:05:53.127737 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet da90061bf3d9471896a2e51bb02864bb (table rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]) successfully deleted
I20240502 08:05:53.195288 19883 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:53.195854 21480 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.199335 19752 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:53.199985 21481 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.201766 20015 consensus_queue.cc:237] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:53.203320 20509 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:53.203768 19883 raft_consensus.cc:1261] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:53.203760 21558 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:53.204098 21558 consensus_queue.cc:1035] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:53.205477 21480 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.205821 20509 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.207798 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ee8e3ede5eb4640bbb530277175dd52 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:53.208096 19883 raft_consensus.cc:2932] T 0ee8e3ede5eb4640bbb530277175dd52 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.208225 19606 catalog_manager.cc:5484] T 0ee8e3ede5eb4640bbb530277175dd52 P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.211438 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee8e3ede5eb4640bbb530277175dd52 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:53.211874 21568 tablet_replica.cc:330] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:53.212033 21568 raft_consensus.cc:2229] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:53.212132 21568 raft_consensus.cc:2258] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:53.212872 21568 ts_tablet_manager.cc:1875] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:53.214795 21568 ts_tablet_manager.cc:1888] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:05:53.214864 21568 log.cc:1196] T 0ee8e3ede5eb4640bbb530277175dd52 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/0ee8e3ede5eb4640bbb530277175dd52
I20240502 08:05:53.215296 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet 0ee8e3ede5eb4640bbb530277175dd52 (table rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]) successfully deleted
I20240502 08:05:53.336447 20015 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:53.337770 20509 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:53.337859 19752 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:53.338320 21558 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.338511 21480 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.340332 21481 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.340765 19752 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.340866 20509 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.342978 19609 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:53.346335 20015 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:53.348081 19752 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:53.348101 20509 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:53.348556 21480 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:53.348783 21481 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 08:05:53.348932 19949 consensus_peers.cc:485] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b -> Peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299): Couldn't send request to peer a665e175a5ac4d2b8bffa107bfd68ec9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:53.350629 21480 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:53.351406 19752 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:53.352294 19597 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b44e8f73540a40d080c94498e12b1148 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:53.353041 19606 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.354259 20509 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:53.459790 20509 raft_consensus.cc:1203] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 08:05:53.570078 19883 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:53.571491 20509 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:53.571552 19752 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:53.572160 21612 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.572405 20234 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:53.574323 21612 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.574767 20509 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.575120 19752 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:53.577343 19609 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:53.581015 19883 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:53.582542 20509 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:53.582542 19752 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:05:53.582753 19820 consensus_peers.cc:485] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 -> Peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147): Couldn't send request to peer 9e4a71b756c544e4ae6c8cbe92b6eb5b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:53.582948 21620 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:53.583196 21612 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:53.585178 20234 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:53.585695 20509 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:53.586158 19752 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:53.587208 19594 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3f8d28c74e4b4be089a0d9cea5331e3c with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:53.588541 19606 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:53.950141 21646 ts_tablet_manager.cc:902] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Initiating tablet copy from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:53.950651 21646 tablet_copy_client.cc:308] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:44147
I20240502 08:05:53.951195 20036 tablet_copy_service.cc:140] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Received BeginTabletCopySession request for tablet b44e8f73540a40d080c94498e12b1148 from peer a665e175a5ac4d2b8bffa107bfd68ec9 ({username='slave'} at 127.16.36.194:49229)
I20240502 08:05:53.951306 20036 tablet_copy_service.cc:161] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Beginning new tablet copy session on tablet b44e8f73540a40d080c94498e12b1148 from peer a665e175a5ac4d2b8bffa107bfd68ec9 at {username='slave'} at 127.16.36.194:49229: session id = a665e175a5ac4d2b8bffa107bfd68ec9-b44e8f73540a40d080c94498e12b1148
I20240502 08:05:53.952021 20036 tablet_copy_source_session.cc:205] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:53.952754 21646 ts_tablet_manager.cc:1875] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:53.954782 21646 ts_tablet_manager.cc:1888] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:53.954939 21646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b44e8f73540a40d080c94498e12b1148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:53.955864 21646 tablet_copy_client.cc:783] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:53.956008 21646 tablet_copy_client.cc:647] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:53.957283 21646 tablet_copy_client.cc:523] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:53.958315 21646 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:53.976198 21646 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:53.976449 21646 tablet_bootstrap.cc:492] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap complete.
I20240502 08:05:53.976644 21646 ts_tablet_manager.cc:1367] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.005s
I20240502 08:05:53.977015 21646 raft_consensus.cc:348] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:53.977123 21646 raft_consensus.cc:729] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: LEARNER
I20240502 08:05:53.977210 21646 consensus_queue.cc:260] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:53.977460 21646 ts_tablet_manager.cc:1398] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:53.978128 20036 tablet_copy_service.cc:342] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Request end of tablet copy session a665e175a5ac4d2b8bffa107bfd68ec9-b44e8f73540a40d080c94498e12b1148 received from {username='slave'} at 127.16.36.194:49229
I20240502 08:05:53.978227 20036 tablet_copy_service.cc:433] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: ending tablet copy session a665e175a5ac4d2b8bffa107bfd68ec9-b44e8f73540a40d080c94498e12b1148 on tablet b44e8f73540a40d080c94498e12b1148 with peer a665e175a5ac4d2b8bffa107bfd68ec9
I20240502 08:05:53.984320 21652 ts_tablet_manager.cc:902] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Initiating tablet copy from peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:53.984846 21652 tablet_copy_client.cc:308] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:45299
I20240502 08:05:53.985265 19904 tablet_copy_service.cc:140] P a665e175a5ac4d2b8bffa107bfd68ec9: Received BeginTabletCopySession request for tablet 3f8d28c74e4b4be089a0d9cea5331e3c from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b ({username='slave'} at 127.16.36.195:52927)
I20240502 08:05:53.985385 19904 tablet_copy_service.cc:161] P a665e175a5ac4d2b8bffa107bfd68ec9: Beginning new tablet copy session on tablet 3f8d28c74e4b4be089a0d9cea5331e3c from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b at {username='slave'} at 127.16.36.195:52927: session id = 9e4a71b756c544e4ae6c8cbe92b6eb5b-3f8d28c74e4b4be089a0d9cea5331e3c
I20240502 08:05:53.986136 19904 tablet_copy_source_session.cc:205] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:53.986775 21652 ts_tablet_manager.cc:1875] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:53.988724 21652 ts_tablet_manager.cc:1888] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:53.988888 21652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f8d28c74e4b4be089a0d9cea5331e3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:53.989771 21652 tablet_copy_client.cc:783] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:53.989914 21652 tablet_copy_client.cc:647] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:53.990954 21652 tablet_copy_client.cc:523] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:53.991847 21652 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:54.011459 21652 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:54.011824 21652 tablet_bootstrap.cc:492] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap complete.
I20240502 08:05:54.012002 21652 ts_tablet_manager.cc:1367] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20240502 08:05:54.012501 21652 raft_consensus.cc:348] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:54.012631 21652 raft_consensus.cc:729] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: LEARNER
I20240502 08:05:54.012737 21652 consensus_queue.cc:260] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:54.013007 21652 ts_tablet_manager.cc:1398] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 08:05:54.013659 19904 tablet_copy_service.cc:342] P a665e175a5ac4d2b8bffa107bfd68ec9: Request end of tablet copy session 9e4a71b756c544e4ae6c8cbe92b6eb5b-3f8d28c74e4b4be089a0d9cea5331e3c received from {username='slave'} at 127.16.36.195:52927
I20240502 08:05:54.013770 19904 tablet_copy_service.cc:433] P a665e175a5ac4d2b8bffa107bfd68ec9: ending tablet copy session 9e4a71b756c544e4ae6c8cbe92b6eb5b-3f8d28c74e4b4be089a0d9cea5331e3c on tablet 3f8d28c74e4b4be089a0d9cea5331e3c with peer 9e4a71b756c544e4ae6c8cbe92b6eb5b
I20240502 08:05:54.050026 21604 raft_consensus.cc:1050] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:54.050453 21604 consensus_queue.cc:237] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:54.051641 20508 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 1 LEARNER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:54.051750 20015 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:54.051770 19883 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 08:05:54.052125 21676 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:54.052327 21675 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:54.052515 21673 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:54.054644 21673 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:54.055065 19883 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:54.055642 20015 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:54.056727 20508 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:54.058305 19609 catalog_manager.cc:5484] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 5 to 6, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:54.256786 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40462
I20240502 08:05:54.256918 19752 raft_consensus.cc:595] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:54.347182 19883 raft_consensus.cc:1203] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:05:54.478724 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40480
I20240502 08:05:54.478858 19752 raft_consensus.cc:595] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:54.527390 20015 raft_consensus.cc:1203] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:05:54.553462 21675 raft_consensus.cc:979] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: : Instructing follower 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:54.553588 21675 raft_consensus.cc:1067] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Signalling peer 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:54.554046 20015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0"
dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
from {username='slave'} at 127.16.36.193:57099
I20240502 08:05:54.554203 20015 raft_consensus.cc:482] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:54.554275 20015 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:54.555198 20015 raft_consensus.cc:504] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:54.555604 20015 leader_election.cc:288] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299), cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635)
I20240502 08:05:54.555905 19752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:54.556002 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:54.556030 19752 raft_consensus.cc:3032] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:54.556082 19752 raft_consensus.cc:729] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:54.556111 19883 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:54.556221 19752 consensus_queue.cc:260] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:54.556401 19752 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:54.556969 19883 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:54.557229 19752 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:54.557246 20508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf541bb2ff945a386808e1b025cfec0" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "cc250aaf098d4d10b71f6aa076a47386"
I20240502 08:05:54.557381 20508 raft_consensus.cc:3037] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:54.557557 19951 leader_election.cc:302] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8ac6fd30b17d47b99c1f0eceb5bddd6f, 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:54.557725 21701 raft_consensus.cc:2781] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:54.557828 21701 raft_consensus.cc:686] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:54.557991 21701 consensus_queue.cc:237] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:54.558283 20508 raft_consensus.cc:2454] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:54.559711 19606 catalog_manager.cc:5484] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 1 to 2, leader changed from 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:54.894474 21481 raft_consensus.cc:1050] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: attempting to promote NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 to VOTER
I20240502 08:05:54.894902 21481 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:54.896236 19883 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:54.896420 19752 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:54.896389 20508 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:54.896656 21743 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:54.896880 21701 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:54.897125 21744 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:54.899308 21743 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.899933 19883 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.901010 19752 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.901957 20508 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.903650 19602 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.907063 20015 consensus_queue.cc:237] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:54.908182 19883 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:54.908252 20508 raft_consensus.cc:1261] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:54.908618 21743 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:54.908970 21744 consensus_queue.cc:1035] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:54.910537 21701 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.911144 20508 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.912076 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b44e8f73540a40d080c94498e12b1148 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:54.913165 19605 catalog_manager.cc:5484] T b44e8f73540a40d080c94498e12b1148 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:54.913373 19883 raft_consensus.cc:2932] T b44e8f73540a40d080c94498e12b1148 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:54.916350 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet b44e8f73540a40d080c94498e12b1148 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:54.916724 21759 tablet_replica.cc:330] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:54.916847 21759 raft_consensus.cc:2229] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:54.917097 21759 raft_consensus.cc:2258] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:54.917834 21759 ts_tablet_manager.cc:1875] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:54.919601 21759 ts_tablet_manager.cc:1888] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:05:54.919684 21759 log.cc:1196] T b44e8f73540a40d080c94498e12b1148 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/b44e8f73540a40d080c94498e12b1148
I20240502 08:05:54.920393 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet b44e8f73540a40d080c94498e12b1148 (table rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]) successfully deleted
I20240502 08:05:54.977237 19883 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:54.977689 21744 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.017190 20508 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:55.017705 21744 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.020695 19752 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:55.021203 21743 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.022212 20015 consensus_queue.cc:237] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:55.023941 19883 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:55.024410 21701 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:55.024596 20508 raft_consensus.cc:1261] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:55.024948 21701 consensus_queue.cc:1035] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:55.026520 21744 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.027189 19883 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.028366 20508 raft_consensus.cc:2932] T 5bf541bb2ff945a386808e1b025cfec0 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.029788 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5bf541bb2ff945a386808e1b025cfec0 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:55.030057 19605 catalog_manager.cc:5484] T 5bf541bb2ff945a386808e1b025cfec0 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 6 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:55.032536 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf541bb2ff945a386808e1b025cfec0 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:55.032790 21759 tablet_replica.cc:330] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:55.032905 21759 raft_consensus.cc:2229] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:55.033027 21759 raft_consensus.cc:2258] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:55.033659 21759 ts_tablet_manager.cc:1875] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:55.035537 21759 ts_tablet_manager.cc:1888] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:05:55.035604 21759 log.cc:1196] T 5bf541bb2ff945a386808e1b025cfec0 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/5bf541bb2ff945a386808e1b025cfec0
I20240502 08:05:55.036108 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet 5bf541bb2ff945a386808e1b025cfec0 (table rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]) successfully deleted
I20240502 08:05:55.115126 21723 raft_consensus.cc:1050] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9: attempting to promote NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b to VOTER
I20240502 08:05:55.115532 21723 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:55.116838 20015 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:55.117137 20508 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:55.117208 19752 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:55.117341 21772 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:55.117693 21771 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:55.117931 21769 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:55.119774 21772 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.120115 20015 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.120688 19752 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.122596 19606 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.123310 20508 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.126554 19883 consensus_queue.cc:237] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:55.127689 20015 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:55.127760 20508 raft_consensus.cc:1261] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:55.128028 21771 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:55.128252 21723 consensus_queue.cc:1035] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:55.129555 21771 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.130002 20015 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.130396 20508 raft_consensus.cc:2932] T 3f8d28c74e4b4be089a0d9cea5331e3c P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:55.131345 19594 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3f8d28c74e4b4be089a0d9cea5331e3c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:55.132018 19609 catalog_manager.cc:5484] T 3f8d28c74e4b4be089a0d9cea5331e3c P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:55.134897 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f8d28c74e4b4be089a0d9cea5331e3c with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:55.135131 21759 tablet_replica.cc:330] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:55.135265 21759 raft_consensus.cc:2229] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:55.135473 21759 raft_consensus.cc:2258] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:55.136422 21759 ts_tablet_manager.cc:1875] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:55.137480 19752 consensus_queue.cc:237] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } }
I20240502 08:05:55.138487 21759 ts_tablet_manager.cc:1888] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:05:55.138557 21759 log.cc:1196] T 3f8d28c74e4b4be089a0d9cea5331e3c P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/3f8d28c74e4b4be089a0d9cea5331e3c
I20240502 08:05:55.138623 20015 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:55.138692 19883 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 08:05:55.139041 21673 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:55.139125 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet 3f8d28c74e4b4be089a0d9cea5331e3c (table rebalance_test_table_0 [id=ed6dbfe6b61946828fcaa32f264b377e]) successfully deleted
I20240502 08:05:55.139209 21675 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 08:05:55.141080 21785 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:55.141521 19883 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:55.141541 20015 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:55.143867 19605 catalog_manager.cc:5484] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } }
I20240502 08:05:55.147532 19752 consensus_queue.cc:237] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:55.149017 20015 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:55.149147 19883 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 08:05:55.149435 21673 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 08:05:55.149775 19687 consensus_peers.cc:485] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c4fb281e443457fac00ceaa3e1df876. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:55.150074 21673 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 08:05:55.151535 21673 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:55.153375 19596 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3c4fb281e443457fac00ceaa3e1df876 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:55.153878 19883 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:55.153887 20015 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } } }
I20240502 08:05:55.155543 19602 catalog_manager.cc:5484] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 4 to 5, NON_VOTER cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:55.157569 19752 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40518
I20240502 08:05:55.157714 19752 raft_consensus.cc:595] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:55.379750 19752 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:55.381119 20015 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:55.381119 20508 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:55.381510 21785 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.381781 21675 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.383370 21785 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.383814 20015 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.385681 19609 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:55.388962 19752 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:55.390426 20015 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 08:05:55.390502 19685 consensus_peers.cc:485] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f -> Peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299): Couldn't send request to peer a665e175a5ac4d2b8bffa107bfd68ec9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:55.390784 21673 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:55.392081 21785 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:55.392585 20015 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:55.393370 19596 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a28c958e9d4493a112424486c56534 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:55.394444 19609 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:55.395430 20509 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:55.396147 21675 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.397228 20509 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.398468 20509 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } } }
I20240502 08:05:55.402808 19883 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:55.403968 20509 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:55.404042 19752 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 08:05:55.404367 21771 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.404558 21772 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:55.406361 21772 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.406908 19752 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.407099 20509 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.408972 19609 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 6 to 7. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:55.412333 19883 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
W20240502 08:05:55.414175 19820 consensus_peers.cc:485] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 -> Peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147): Couldn't send request to peer 9e4a71b756c544e4ae6c8cbe92b6eb5b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:55.414201 19752 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:55.414225 20509 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 08:05:55.414573 21771 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:55.414940 21771 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:55.416386 21772 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:55.416772 20509 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:55.416831 19751 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } } }
I20240502 08:05:55.418095 19594 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 179b4d44bff84b84b74d66c98a506f50 with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 08:05:55.418766 19605 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 7 to 8, NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) added. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:55.563184 21673 raft_consensus.cc:979] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: : Instructing follower 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:55.563302 21673 raft_consensus.cc:1067] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Signalling peer 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:55.563727 20015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876"
dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
from {username='slave'} at 127.16.36.193:57099
I20240502 08:05:55.563872 20015 raft_consensus.cc:482] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:55.563938 20015 raft_consensus.cc:3037] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:55.564872 20015 raft_consensus.cc:504] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:55.565266 20015 leader_election.cc:288] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:55.565622 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:55.565649 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4fb281e443457fac00ceaa3e1df876" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:55.565759 19883 raft_consensus.cc:3037] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:55.565758 19751 raft_consensus.cc:3032] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:55.565807 19751 raft_consensus.cc:729] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:55.565976 19751 consensus_queue.cc:260] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:55.566176 19751 raft_consensus.cc:3037] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:55.566812 19883 raft_consensus.cc:2454] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:55.567165 19751 raft_consensus.cc:2454] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:55.567179 19949 leader_election.cc:302] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9; no voters:
I20240502 08:05:55.567467 21701 raft_consensus.cc:2781] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:55.567648 21701 raft_consensus.cc:686] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:55.567811 21701 consensus_queue.cc:237] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:55.569489 19609 catalog_manager.cc:5484] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 1 to 2, leader changed from 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:55.877274 21869 ts_tablet_manager.cc:902] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Initiating tablet copy from peer a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:55.877832 21869 tablet_copy_client.cc:308] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.194:45299
I20240502 08:05:55.878387 19904 tablet_copy_service.cc:140] P a665e175a5ac4d2b8bffa107bfd68ec9: Received BeginTabletCopySession request for tablet 179b4d44bff84b84b74d66c98a506f50 from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b ({username='slave'} at 127.16.36.195:52927)
I20240502 08:05:55.878509 19904 tablet_copy_service.cc:161] P a665e175a5ac4d2b8bffa107bfd68ec9: Beginning new tablet copy session on tablet 179b4d44bff84b84b74d66c98a506f50 from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b at {username='slave'} at 127.16.36.195:52927: session id = 9e4a71b756c544e4ae6c8cbe92b6eb5b-179b4d44bff84b84b74d66c98a506f50
I20240502 08:05:55.879321 19904 tablet_copy_source_session.cc:205] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:55.880169 21869 ts_tablet_manager.cc:1875] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:55.882618 21869 ts_tablet_manager.cc:1888] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:55.882829 21869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179b4d44bff84b84b74d66c98a506f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:55.883984 21869 tablet_copy_client.cc:783] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:55.884127 21869 tablet_copy_client.cc:647] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:55.885404 21869 tablet_copy_client.cc:523] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:55.886593 21869 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap starting.
I20240502 08:05:55.906384 21869 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:55.906721 21869 tablet_bootstrap.cc:492] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Bootstrap complete.
I20240502 08:05:55.906935 21869 ts_tablet_manager.cc:1367] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240502 08:05:55.907377 21869 raft_consensus.cc:348] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:55.907493 21869 raft_consensus.cc:729] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Initialized, Role: LEARNER
I20240502 08:05:55.907595 21869 consensus_queue.cc:260] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: NON_VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: true } }
I20240502 08:05:55.907872 21869 ts_tablet_manager.cc:1398] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:55.908561 19904 tablet_copy_service.cc:342] P a665e175a5ac4d2b8bffa107bfd68ec9: Request end of tablet copy session 9e4a71b756c544e4ae6c8cbe92b6eb5b-179b4d44bff84b84b74d66c98a506f50 received from {username='slave'} at 127.16.36.195:52927
I20240502 08:05:55.908645 19904 tablet_copy_service.cc:433] P a665e175a5ac4d2b8bffa107bfd68ec9: ending tablet copy session 9e4a71b756c544e4ae6c8cbe92b6eb5b-179b4d44bff84b84b74d66c98a506f50 on tablet 179b4d44bff84b84b74d66c98a506f50 with peer 9e4a71b756c544e4ae6c8cbe92b6eb5b
I20240502 08:05:55.940572 21875 ts_tablet_manager.cc:902] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Initiating tablet copy from peer 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001)
I20240502 08:05:55.941179 21875 tablet_copy_client.cc:308] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.193:36001
I20240502 08:05:55.941769 19772 tablet_copy_service.cc:140] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Received BeginTabletCopySession request for tablet 12a28c958e9d4493a112424486c56534 from peer a665e175a5ac4d2b8bffa107bfd68ec9 ({username='slave'} at 127.16.36.194:38087)
I20240502 08:05:55.941884 19772 tablet_copy_service.cc:161] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Beginning new tablet copy session on tablet 12a28c958e9d4493a112424486c56534 from peer a665e175a5ac4d2b8bffa107bfd68ec9 at {username='slave'} at 127.16.36.194:38087: session id = a665e175a5ac4d2b8bffa107bfd68ec9-12a28c958e9d4493a112424486c56534
I20240502 08:05:55.942667 19772 tablet_copy_source_session.cc:205] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:55.943431 21875 ts_tablet_manager.cc:1875] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 08:05:55.945417 21875 ts_tablet_manager.cc:1888] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240502 08:05:55.945575 21875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a28c958e9d4493a112424486c56534. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:55.946504 21875 tablet_copy_client.cc:783] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:55.946625 21875 tablet_copy_client.cc:647] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:55.947840 21875 tablet_copy_client.cc:523] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:55.948951 21875 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap starting.
I20240502 08:05:55.968190 21875 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:55.968544 21875 tablet_bootstrap.cc:492] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Bootstrap complete.
I20240502 08:05:55.968729 21875 ts_tablet_manager.cc:1367] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.006s
I20240502 08:05:55.969170 21875 raft_consensus.cc:348] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:55.969288 21875 raft_consensus.cc:729] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a665e175a5ac4d2b8bffa107bfd68ec9, State: Initialized, Role: LEARNER
I20240502 08:05:55.969367 21875 consensus_queue.cc:260] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: NON_VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: true } }
I20240502 08:05:55.969627 21875 ts_tablet_manager.cc:1398] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 08:05:55.970290 19772 tablet_copy_service.cc:342] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Request end of tablet copy session a665e175a5ac4d2b8bffa107bfd68ec9-12a28c958e9d4493a112424486c56534 received from {username='slave'} at 127.16.36.194:38087
I20240502 08:05:55.970373 19772 tablet_copy_service.cc:433] P 8ac6fd30b17d47b99c1f0eceb5bddd6f: ending tablet copy session a665e175a5ac4d2b8bffa107bfd68ec9-12a28c958e9d4493a112424486c56534 on tablet 12a28c958e9d4493a112424486c56534 with peer a665e175a5ac4d2b8bffa107bfd68ec9
W20240502 08:05:56.005050 19951 consensus_peers.cc:485] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b -> Peer cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635): Couldn't send request to peer cc250aaf098d4d10b71f6aa076a47386. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c4fb281e443457fac00ceaa3e1df876. This is attempt 1: this message will repeat every 5th retry.
I20240502 08:05:56.099709 19751 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:56.100189 21701 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:56.103741 19883 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 08:05:56.104210 21898 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 08:05:56.378293 19883 raft_consensus.cc:1203] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:05:56.385231 20015 raft_consensus.cc:1203] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 08:05:56.388125 21919 ts_tablet_manager.cc:902] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: Initiating tablet copy from peer 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195:44147)
I20240502 08:05:56.388648 21919 tablet_copy_client.cc:308] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Beginning tablet copy session from remote peer at address 127.16.36.195:44147
I20240502 08:05:56.389084 20036 tablet_copy_service.cc:140] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Received BeginTabletCopySession request for tablet 3c4fb281e443457fac00ceaa3e1df876 from peer cc250aaf098d4d10b71f6aa076a47386 ({username='slave'} at 127.16.36.196:44923)
I20240502 08:05:56.389185 20036 tablet_copy_service.cc:161] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Beginning new tablet copy session on tablet 3c4fb281e443457fac00ceaa3e1df876 from peer cc250aaf098d4d10b71f6aa076a47386 at {username='slave'} at 127.16.36.196:44923: session id = cc250aaf098d4d10b71f6aa076a47386-3c4fb281e443457fac00ceaa3e1df876
I20240502 08:05:56.389905 20036 tablet_copy_source_session.cc:205] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 08:05:56.390750 21919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4fb281e443457fac00ceaa3e1df876. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 08:05:56.392509 21919 tablet_copy_client.cc:783] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 0 data blocks...
I20240502 08:05:56.392728 21919 tablet_copy_client.cc:647] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Starting download of 1 WAL segments...
I20240502 08:05:56.393883 21919 tablet_copy_client.cc:523] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 08:05:56.394901 21919 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap starting.
I20240502 08:05:56.414032 21919 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 08:05:56.414333 21919 tablet_bootstrap.cc:492] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: Bootstrap complete.
I20240502 08:05:56.414527 21919 ts_tablet_manager.cc:1367] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: Time spent bootstrapping tablet: real 0.020s user 0.018s sys 0.000s
I20240502 08:05:56.415007 21919 raft_consensus.cc:348] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:56.415143 21919 raft_consensus.cc:729] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cc250aaf098d4d10b71f6aa076a47386, State: Initialized, Role: LEARNER
I20240502 08:05:56.415244 21919 consensus_queue.cc:260] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: NON_VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: true } }
I20240502 08:05:56.415635 21919 ts_tablet_manager.cc:1398] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240502 08:05:56.416276 20036 tablet_copy_service.cc:342] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: Request end of tablet copy session cc250aaf098d4d10b71f6aa076a47386-3c4fb281e443457fac00ceaa3e1df876 received from {username='slave'} at 127.16.36.196:44923
I20240502 08:05:56.416373 20036 tablet_copy_service.cc:433] P 9e4a71b756c544e4ae6c8cbe92b6eb5b: ending tablet copy session cc250aaf098d4d10b71f6aa076a47386-3c4fb281e443457fac00ceaa3e1df876 on tablet 3c4fb281e443457fac00ceaa3e1df876 with peer cc250aaf098d4d10b71f6aa076a47386
I20240502 08:05:56.791103 21673 raft_consensus.cc:1050] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: attempting to promote NON_VOTER a665e175a5ac4d2b8bffa107bfd68ec9 to VOTER
I20240502 08:05:56.791508 21673 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:56.792865 20015 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:56.792922 20509 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:56.793283 19883 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEARNER]: Refusing update from remote peer 8ac6fd30b17d47b99c1f0eceb5bddd6f: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:56.793298 21957 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:56.793563 21673 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:56.793710 21917 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:56.795606 21673 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:56.796183 20015 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:56.798027 20509 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:56.798542 19883 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:56.798837 19606 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f reported cstate change: config changed from index 8 to 9, a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:56.956862 19751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12a28c958e9d4493a112424486c56534"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40624
I20240502 08:05:56.957015 19751 raft_consensus.cc:595] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:56.974488 21918 raft_consensus.cc:1050] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9: attempting to promote NON_VOTER 9e4a71b756c544e4ae6c8cbe92b6eb5b to VOTER
I20240502 08:05:56.974920 21918 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:56.976217 20015 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 LEARNER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:56.976284 20509 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:56.976281 19751 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 08:05:56.976666 21988 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:56.976891 21986 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:56.977172 21985 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 08:05:56.979000 21985 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.979460 19751 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.979593 20509 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.981948 19609 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.982708 20015 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.983916 20509 raft_consensus.cc:1203] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 LEARNER]: Deduplicated request from leader. Original: 1.5->[2.6-2.6] Dedup: 2.6->[]
I20240502 08:05:56.986452 19883 consensus_queue.cc:237] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } }
I20240502 08:05:56.987838 20509 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:56.988301 21986 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:56.988340 20015 raft_consensus.cc:1261] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Refusing update from remote peer a665e175a5ac4d2b8bffa107bfd68ec9: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 08:05:56.988696 21985 consensus_queue.cc:1035] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:56.989960 21986 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.990383 20509 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.990693 20015 raft_consensus.cc:2932] T 179b4d44bff84b84b74d66c98a506f50 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.992151 19594 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 179b4d44bff84b84b74d66c98a506f50 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:56.992511 19609 catalog_manager.cc:5484] T 179b4d44bff84b84b74d66c98a506f50 P cc250aaf098d4d10b71f6aa076a47386 reported cstate change: config changed from index 9 to 10, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 1 leader_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } attrs { promote: false } } }
I20240502 08:05:56.994951 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 179b4d44bff84b84b74d66c98a506f50 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:56.995492 21992 tablet_replica.cc:330] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:56.995683 21992 raft_consensus.cc:2229] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:56.995896 21992 raft_consensus.cc:2258] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:56.996603 21992 ts_tablet_manager.cc:1875] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:56.998580 21992 ts_tablet_manager.cc:1888] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 08:05:56.998646 21992 log.cc:1196] T 179b4d44bff84b84b74d66c98a506f50 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/179b4d44bff84b84b74d66c98a506f50
I20240502 08:05:56.999171 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet 179b4d44bff84b84b74d66c98a506f50 (table rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]) successfully deleted
I20240502 08:05:57.180081 19751 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "12a28c958e9d4493a112424486c56534"
dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40632
I20240502 08:05:57.180233 19751 raft_consensus.cc:595] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Received request to transfer leadership
I20240502 08:05:57.184669 21917 raft_consensus.cc:979] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: : Instructing follower 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:57.184792 21917 raft_consensus.cc:1067] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Signalling peer 9e4a71b756c544e4ae6c8cbe92b6eb5b to start an election
I20240502 08:05:57.185232 20015 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "12a28c958e9d4493a112424486c56534"
dest_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b"
from {username='slave'} at 127.16.36.193:57099
I20240502 08:05:57.185413 20015 raft_consensus.cc:482] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 08:05:57.185477 20015 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.186700 20015 raft_consensus.cc:504] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:57.187223 20015 leader_election.cc:288] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Requested vote from peers 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001), cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196:35635), a665e175a5ac4d2b8bffa107bfd68ec9 (127.16.36.194:45299)
I20240502 08:05:57.187633 20509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "cc250aaf098d4d10b71f6aa076a47386"
I20240502 08:05:57.187633 19751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f"
I20240502 08:05:57.187762 19751 raft_consensus.cc:3032] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Stepping down as leader of term 1
I20240502 08:05:57.187716 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a28c958e9d4493a112424486c56534" candidate_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: true dest_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9"
I20240502 08:05:57.187769 20509 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.187815 19751 raft_consensus.cc:729] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8ac6fd30b17d47b99c1f0eceb5bddd6f, State: Running, Role: LEADER
I20240502 08:05:57.187855 19883 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.187973 19751 consensus_queue.cc:260] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:57.188185 19751 raft_consensus.cc:3037] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 1 FOLLOWER]: Advancing to term 2
I20240502 08:05:57.188827 19883 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:57.188856 20509 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:57.189222 19751 raft_consensus.cc:2454] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e4a71b756c544e4ae6c8cbe92b6eb5b in term 2.
I20240502 08:05:57.189360 19951 leader_election.cc:302] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9e4a71b756c544e4ae6c8cbe92b6eb5b, a665e175a5ac4d2b8bffa107bfd68ec9, cc250aaf098d4d10b71f6aa076a47386; no voters:
I20240502 08:05:57.189574 21896 raft_consensus.cc:2781] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 FOLLOWER]: Leader election won for term 2
I20240502 08:05:57.189669 21896 raft_consensus.cc:686] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Becoming Leader. State: Replica: 9e4a71b756c544e4ae6c8cbe92b6eb5b, State: Running, Role: LEADER
I20240502 08:05:57.189815 21896 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:57.191779 19602 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: term changed from 1 to 2, leader changed from 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) to 9e4a71b756c544e4ae6c8cbe92b6eb5b (127.16.36.195). New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 08:05:57.475818 21896 raft_consensus.cc:1050] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b: attempting to promote NON_VOTER cc250aaf098d4d10b71f6aa076a47386 to VOTER
I20240502 08:05:57.476377 21896 consensus_queue.cc:237] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:57.478000 20509 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 LEARNER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:57.478081 19751 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:57.478394 19882 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 08:05:57.478577 21898 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:57.478773 22024 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:57.479036 21701 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 08:05:57.481899 22024 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.482275 19751 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.482301 19882 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.484987 19605 catalog_manager.cc:5484] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 5 to 7, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.485338 20509 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, cc250aaf098d4d10b71f6aa076a47386 (127.16.36.196) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.488595 20015 consensus_queue.cc:237] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } }
I20240502 08:05:57.489723 19882 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:57.489810 20509 raft_consensus.cc:1261] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 08:05:57.490100 21898 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:57.490500 21898 consensus_queue.cc:1035] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 08:05:57.492086 21701 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.492722 19882 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.492708 20509 raft_consensus.cc:2932] T 3c4fb281e443457fac00ceaa3e1df876 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } }
I20240502 08:05:57.493731 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c4fb281e443457fac00ceaa3e1df876 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:57.495072 19606 catalog_manager.cc:5484] T 3c4fb281e443457fac00ceaa3e1df876 P 9e4a71b756c544e4ae6c8cbe92b6eb5b reported cstate change: config changed from index 7 to 8, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 08:05:57.497591 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4fb281e443457fac00ceaa3e1df876 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:57.497848 21992 tablet_replica.cc:330] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:57.497954 21992 raft_consensus.cc:2229] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:57.498080 21992 raft_consensus.cc:2258] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:57.498764 21992 ts_tablet_manager.cc:1875] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:57.500602 21992 ts_tablet_manager.cc:1888] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240502 08:05:57.500687 21992 log.cc:1196] T 3c4fb281e443457fac00ceaa3e1df876 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/3c4fb281e443457fac00ceaa3e1df876
I20240502 08:05:57.501214 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet 3c4fb281e443457fac00ceaa3e1df876 (table rebalance_test_table_2 [id=590bbdab6e5549e28e350ce8d2d2ccb9]) successfully deleted
I20240502 08:05:57.589455 19751 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:57.589982 22024 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac6fd30b17d47b99c1f0eceb5bddd6f" member_type: VOTER last_known_addr { host: "127.16.36.193" port: 36001 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:57.710783 19882 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:57.711195 22024 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:57.714099 20509 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 08:05:57.714936 21701 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 08:05:58.276031 20015 consensus_queue.cc:237] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } }
I20240502 08:05:58.277251 20509 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:05:58.277290 19882 raft_consensus.cc:1261] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Refusing update from remote peer 9e4a71b756c544e4ae6c8cbe92b6eb5b: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 08:05:58.277670 22058 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:58.277897 21701 consensus_queue.cc:1035] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 08:05:58.279495 22057 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P 9e4a71b756c544e4ae6c8cbe92b6eb5b [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:58.279901 20509 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P cc250aaf098d4d10b71f6aa076a47386 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:58.279928 19882 raft_consensus.cc:2932] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:58.281941 19597 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12a28c958e9d4493a112424486c56534 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 08:05:58.282495 19609 catalog_manager.cc:5484] T 12a28c958e9d4493a112424486c56534 P a665e175a5ac4d2b8bffa107bfd68ec9 reported cstate change: config changed from index 9 to 11, VOTER 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193) evicted. New cstate: current_term: 2 leader_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "9e4a71b756c544e4ae6c8cbe92b6eb5b" member_type: VOTER last_known_addr { host: "127.16.36.195" port: 44147 } } peers { permanent_uuid: "cc250aaf098d4d10b71f6aa076a47386" member_type: VOTER last_known_addr { host: "127.16.36.196" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "a665e175a5ac4d2b8bffa107bfd68ec9" member_type: VOTER last_known_addr { host: "127.16.36.194" port: 45299 } attrs { promote: false } } }
I20240502 08:05:58.285229 19732 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a28c958e9d4493a112424486c56534 with delete_type TABLET_DATA_TOMBSTONED (TS 8ac6fd30b17d47b99c1f0eceb5bddd6f not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:50760
I20240502 08:05:58.285585 22063 tablet_replica.cc:330] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: stopping tablet replica
I20240502 08:05:58.285722 22063 raft_consensus.cc:2229] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 08:05:58.285848 22063 raft_consensus.cc:2258] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 08:05:58.286468 22063 ts_tablet_manager.cc:1875] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 08:05:58.288439 22063 ts_tablet_manager.cc:1888] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.10
I20240502 08:05:58.288558 22063 log.cc:1196] T 12a28c958e9d4493a112424486c56534 P 8ac6fd30b17d47b99c1f0eceb5bddd6f: Deleting WAL directory at /tmp/dist-test-taskm5AjVx/test-tmp/rebalancer_tool-test.2.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx1_rf3_t3_l_3_1.1714636688671911-16531-0/raft_consensus-itest-cluster/ts-0/wal/wals/12a28c958e9d4493a112424486c56534
I20240502 08:05:58.289165 19596 catalog_manager.cc:4830] TS 8ac6fd30b17d47b99c1f0eceb5bddd6f (127.16.36.193:36001): tablet 12a28c958e9d4493a112424486c56534 (table rebalance_test_table_1 [id=cfd6656d59524291be19d399c8b25d18]) successfully deleted
I20240502 08:05:58.703377 19995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:58.704448 19732 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.707201 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.715974 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:58.793133 19732 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.794296 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:58.794780 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.796093 19995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:58.882261 19995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:58.882542 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:58.883857 19732 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.883919 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.970788 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:58.971448 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.977032 19732 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:58.984798 19995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:59.057752 19732 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:59.059947 19995 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 08:05:59.068749 19863 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 08:05:59.069859 20490 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 08:05:59.105291 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 19670
I20240502 08:05:59.112428 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 19801
W20240502 08:05:59.120849 19949 connection.cc:535] client connection to 127.16.36.194:45299 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 08:05:59.121048 19949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 08:05:59.121376 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 19933
I20240502 08:05:59.128720 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 20428
I20240502 08:05:59.135003 16531 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskm5AjVx/build/debug/bin/kudu with pid 19579
2024-05-02T08:05:59Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx1_rf3_t3_l_3_1 (19566 ms)
[----------] 2 tests from LocationAwareRebalancingParamTest (35440 ms total)
[----------] Global test environment tear-down
[==========] 7 tests from 6 test suites ran. (470480 ms total)
[ PASSED ] 6 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/0, where GetParam() = 4-byte object <00-00 00-00>
1 FAILED TEST